Data flow
Data Preparation is the Dattos platform's ETL module. In it, you collect, clean, transform, combine, and organize information from different systems and external sources to generate reliable results for analysis, BI, and visualizations. All this in a no-code interface, designed for business users, which replaces traditional spreadsheets with more agility and security.
This article introduces the flow designer: the screen areas, how to build and configure a flow, and the features that help document and validate the result.
How a preparation flow works
A data flow is a set of connected steps, each performing a function on the information. You add, connect, and configure the steps according to the flow's objective.
Each step can have one or more input and output ports, which is where connections between steps are created. Connections represent the transitions between steps and follow a horizontal logic, from left to right.
The designer header
At the top of the screen, the process name and, from left to right, the action buttons appear:
Close: closes flow editing and takes you to the process View/Execution screen, where the configuration information is located.
Gallery: opens ready-to-use flow templates, with search and filters by data source type and result.
Identification: opens the identification rules for files used in the flow for automatic upload.
Run: takes you to the process Import and Execution screen, considering the flow created in the designer.
Options: allows you to Restore template (recovers a template in Dattos .XDTT format) and Download template (downloads the built flow in the .XDTT standard).
Question mark: opens the System Manual ("Introduction to the Data Preparation Flow" guide) in a new tab and the Start Tutorial, with a step-by-step of the main features and a video on how to configure a flow from scratch.
The toolbar and step categories
The toolbar lists all available steps, grouped by category and identified by color:
Data Source: importing files and data.
Preparation: cleaning, normalization, and data modeling.
Combination: joining and merging data.
Aggregation: consolidation and grouping of data.
Output: exporting data and results.
Adding, connecting, and removing steps
To add: drag the step from the toolbar to the canvas right below.
To connect: bring the step close to another that has a free output and the platform suggests the connection; just release. To connect manually, click on the output port of one step, drag the pointer to the input port of the other and release.
To remove: select the step and click the red x displayed above it, or press the Delete key. Confirm in the window that appears by clicking Confirm.
Configuring a step
To open a step's configuration area, double-click it after adding it to the flow. Options vary depending on the chosen step. Details for each are in the Data Preparation step articles.
Tip: Data Source category steps open the configuration area automatically when they are added to the flow.
Automatic reprocessing: the flow is reprocessed whenever a step is added, edited, or deleted, so the data sample accurately reflects the current result.
Step context menu
Right-click on any step in the flow to open the context menu:
Configure Step: accesses specific settings for adjustments.
Edit Name: personalizes the step's display name for more intuitive identification.
Add Description: inserts information about the step. When a description already exists, the menu offers Edit name and description and Delete description.
Delete Step: removes the step from the flow.
Tip: describing steps makes it easier to understand the flow, saves time, reduces the chance of error, and provides extra clarity in team communication.
Explanatory notes in the designer
In addition to per-step descriptions, you can add freehand notes in the designer to document the flow. Right-click on any blank space and select Add annotation. A window opens for you to include useful information and insights at the most relevant points in the flow.
To edit or delete an annotation, right-click on it and use the options displayed.
Validate by Data Sample
The Data Sample is located in a blue bar at the bottom of the designer. Double-click to expand it. With a step configured and selected, it displays the first 100 rows of that flow step.
The sample window contains:
Output port tabs: available depending on the selected step.
General size information: for example, "Displaying sample of 100 rows out of a total of 300 rows - 20 columns".
Data sample table.
Search field: locates content within the sample.
Simulation date.
Download: exports the sample as an .xlsx file.
Sample limit: the Data Sample displays a maximum of 100 rows, regardless of the size of the data sources loaded in the flow.
Message History
To understand errors and alerts, open the history by clicking any icon in the bottom right corner of the designer. Messages are classified into three types, which can be filtered in the window:
Error: invalid step configurations.
Warning: data flow synchronization failures.
Message: step configuration editing and synchronization events.
The table lists, for each record:
Type: the message category.
Source: the step that originated the message. When not specified, it is an event of the flow as a whole.
Message: the message text.
Template Gallery
The Gallery brings together ready-made flow templates, which you can use in full or in parts. Click the Gallery button in the top right corner to open the selection window, with search and filters by step category and by use case (accounting, tax, financial, health, among others).
By clicking Use Template, the platform loads the template into the designer. To see more details before choosing, click on the area of the desired template to open the screen with the filter tag, a screenshot of the template in the designer, and the description.
Tip: you can add more than one template to your process and then edit the flow to adapt it to your needs.
Configure the Identification Rule
The Identification Rule allows the platform to automatically direct each file sent through the "File Upload" and "Process Execution" screens to the correct flow. The rule uses the content of specific columns to perform this routing.
Click the Identification Rule button to open the screen where you configure the rule for each file-type Data Source. Each source displays a status:
Valid: the input step already has an identification rule configured.
Invalid: the input step does not have a rule configured yet.
To create the rule, select the content of a cell (column and row intersection) and click Add Rule. One or more rules can be defined. Upon completion, the source status changes to Valid.
Tip: keep the Ctrl key pressed to select more than one spreadsheet cell.
Which steps appear: only Data Sources that read files and have a data sample are included in the list. When there are eligible steps without a rule configured yet, a red indicator appears on the Identification Rule button in the designer.
Versioning and restoration
Through the Options action in the designer, you download the flow template (.XDTT) and restore a preparation from a saved file. The full step-by-step is in the Versioning and restoration article.
Related articles
Excel File Step
OFX File Step
Text File Step
Versioning and restoration