Text File Step
The Text File step imports various text files, such as TXT, CSV, DAT, EDI, RET, REM, and BDP, to start a Data Preparation flow using the data in the file. It reads both files with delimiter characters (such as CSV files) and fixed-width files, in which each field is determined by the starting and ending character positions.
Text File is located in the Data Source category on the toolbar. After completing the flow, you upload the files through the Process Preview or File Upload screen on the Dattos platform.
Import the text file
There are two ways to load the file:
Import File: opens the file explorer so you can select the file.
Drag and drop: drag the file directly onto the screen.
Validation during import: only files with valid extensions are accepted, with a maximum size of 100 MB in a Turbo AI process and 50 MB in a Standard process.
Configure the step
On the configuration screen, you will find:
File information: identifies the name of the imported file.
Change File: replaces the imported file. When you import a new file, the platform lets you keep the settings already configured and update only the sample, or update all settings based on the new file.
Name: identifies the name of the step. By default, it uses the name of the imported sample file, but the field can be edited.
In Import Options:
Import data starting from row: enter the row number from which the import should begin bringing in data.
Contains header?: select this option if the file has a row containing the column names.
Under Map Columns, select the information structure pattern for the file: Single record structure or Multiple record structures.
Tip: choose a name that represents the information in the file and is easy for you and your colleagues to understand.
How to choose the structure: use Single record structure when the file has the same column pattern on every row. Use Multiple record structures when the column pattern changes on some rows.
Single record structure
Select the column delimiter, that is, how the columns are separated in the file. The options are: Fixed width, Comma ( , ), Semicolon ( ; ), Colon ( : ), Equals sign ( = ), Pipe ( | ), and Tab (tab).
When you choose Fixed width, enter the ending positions of each column in the format 5, 10, 12. This tells the platform that the first column runs from position 1 to 5, the second from 6 to 10, and the third from 11 to 12.
Adjust the columns
Once the delimiter is defined, adjust each output column:
Column Visibility: clear the selection to disregard the column.
Column Order: shows the order of the columns in the output.
Column Name: shows all columns in the output; the names can be renamed.
Data Type: Alphanumeric, Numeric, or Date. For Date, enter the format mask used by the column. The field can be edited.
Sample: displays the content of the columns from the first imported row.
Use Close to exit without saving and return to the Designer, or Save to save the settings.
Advanced settings
File content encoding standard: Unicode (UTF-8) or Windows (ANSI).
Decimal separator for converting text to numbers: Comma ( , ) or Period ( . ), according to the separator used in the file. For example, content "100,99" read as a number requires the Comma separator; "100.99" requires the Period separator. In both cases, the displayed result is "100,99".
Add the data source name as a column: adds a column containing the step name to the sample.
Add the file name as a column: adds a column containing the name of the imported file to the sample.
Multiple record structures
When the file combines different column patterns in separate rows, select the column delimiter from the same options: Fixed width, Comma ( , ), Semicolon ( ; ), Colon ( : ), Equals sign ( = ), Pipe ( | ), and Tab (tab).
When you choose Fixed width:
Fill in Select record type position with the starting and ending positions separated by a comma, in the format
1, 2, indicating where the record type identifier is located.For each record type found (represented by tabs), fill in Enter the ending position of each column in the format
5, 10, 12. This means that the first column runs from position 1 to 5, the second from 6 to 10, and the third from 11 to 12.
When you choose a character delimiter (comma, semicolon, colon, equals sign, pipe, or tab):
Ignore double quotes as a field delimiter: when selected, the platform ignores the quotation marks and does not consider them part of the field value; when cleared, the quotation marks are included as the value.
Select record type position: enter the number of the column that identifies the record type. For example, "1" indicates that the value differentiating the record type is in the first column.
Adjust the columns by record type
Under Map Columns, configure the columns for each record type as needed by changing the name or data type, or disregarding columns for subsequent steps:
Select record type position: enter the number of the column that indicates the position of the record type.
Select the record: choose which record type you want to configure from those identified in the file.
Column Visibility: clear the selection to disregard the column.
Column Order: shows the order of the columns in the output.
Column Name: shows all columns in the output; the names can be renamed.
Data Type: Alphanumeric, Numeric, or Date. For Date, enter the format mask used by the column.
Sample: displays the content of the columns from the first imported row.
Output configuration
Define the desired outputs by indicating which records are headers and what the content is:
Data Output Name: enter the desired name for the output.
Structure - Header: indicate which records are headers. The order matters because it defines the order of the columns created: the columns follow the order in which the headers are registered.
Structure - Content: define only one record as the information content.
Add New Output: creates an additional data output.
Use Close to exit without saving and return to the Designer, or Save to save the settings.
Advanced settings
File content encoding standard: Unicode (UTF-8) or Windows (ANSI).
Decimal separator for converting text to numbers: Comma ( , ) or Period ( . ), according to the separator used in the file, following the same logic as the single structure.
Add the data source name as a column: adds a column containing the step name to the sample.
Add the file name as a column: adds a column containing the name of the imported file to the sample.
Reference date taken from the file
In Turbo AI processes, the reference date can come from a date column in the text file: each distinct date generates its own load, containing only the rows for that date. The column must be of the Date type, with the correct date format, because a date read incorrectly results in an incorrect reference date.
This setting is not configured in this step, but through the Reference date button in the designer header. The step-by-step instructions are available in the Data flow article.
Related articles
Configure Data Preparation
Excel File Step
OFX File Step