Academy
Transformation

Deduplicate Step

The Deduplicate step identifies whether a record is unique or a duplicate within a data preparation flow. Use it when you need to separate unique occurrences from repeated ones, based on the columns that define what counts as a duplicate.

The Deduplicate step is located in the Preparation category of the toolbar.

Configure Deduplicate

Configuring Deduplicate is simple. As soon as another step is connected to its input, all columns are displayed. Just select the columns that are part of the duplicate validation rule.

  • Column Selection: marks one or more columns that are part of the duplicate rule. By default, all columns are selected.

  • Column Status: indicates the column status, which can be Valid or Invalid. A column becomes Invalid when a column used in the rule is deleted in the input step or has its data type changed. In this status, the platform allows the column to be deleted.

  • Column: shows all columns loaded from the input step. The values in this column are not editable.

  • Data Type: the data type of the input column, which can be Alphanumeric, Numeric, or Date.

  • Close: closes the screen without saving the configurations and returns to the Designer.

  • Save: saves the configurations.

Attention: the step displays an error notice when there is not at least one transition connected.

Output ports

After configuration, Deduplicate has two outputs:

  • Output U: contains all non-duplicate data and the first occurrence of each duplicate record.

  • Output D: contains the other occurrences of each duplicate record.

Configuration example

Consider the input columns Business, Date, Account, Value, and Description, with four records where Account 1001 is repeated in two of them.

By configuring the duplicate rule by the Account column, the result is divided as follows:

  • Output U receives the first occurrence of Account 1001 plus the non-duplicate records (Accounts 1002 and 1004), totaling three records.

  • Output D receives the second occurrence of Account 1001 (Business B, Date 11/14/2022, Value 120, Description "Expenses 2").

  • Transform Step

  • Sort Step

  • Stack Step

Was this helpful?
Was this helpful?