Remove Duplicates
The Remove duplicates step returns a list with repeats taken out. You say what makes two entries the same, and it keeps one of each.
- Use it to
- Deduplicate a list of records by a field name or a list of plain values, keeping the first or last copy
- Find it
- Tools → Data → Lists → Remove Duplicates
- Cost
- 1 Step credit per execution
- Limits
- Up to 100,000 items · Preserves the order items first appeared
- Output
- {{ N.results }} (deduplicated list), {{ N.resultCount }}
- Instead
- Inline pipe ({{ 1.tags | remove_duplicates }}) for flat lists
Use it after combining sources that overlap. The same customer exported from two systems, an address on several orders, a lead that came in twice.
Deduplicating inside a single field? You can remove duplicates directly inside field tokens using Data Transformation with {{ 2.emails | remove_duplicates }} (or {{ 2.records | pluck:email | remove_duplicates }}). Use this dedicated Remove duplicates step when deduplicating complex multi-field records or processing large datasets (up to 100,000 items).
Setting it up
Choose the list
Pick it from the lists the field offers, which come from the steps before this one. Choose Enter it myself to type a reference such as {{ 3.results }} or paste a JSON list instead.
Say what makes two entries the same
Under Compare by field, leave the field empty to compare each plain value directly, or enter a field name (such as email or customer_id) when working with records. Every item in the list must contain the chosen field.
Choose which copy to keep
Under Which copy to keep, choose First or Last.
Which copy you keep changes the data, not the order
The surviving items stay in the order they first appeared, whichever copy wins. Keeping the last one does not reverse anything.
What changes is which record survives, and with real data that matters. Say a customer appears twice, once with an old address and once with a new one. First keeps the old, Last keeps the new, assuming the newer record came later in the list.
For a list of plain values it makes no difference at all: the same value is the same value.
Capital letters count unless you say otherwise
Ignore capitalisation is off to begin with, so Active and active are two different values and both survive.
Turn it on when the same thing is spelled differently by two systems: one exports upper case, the other lower.
What it passes on
The list with repeats removed:
{{ 5.results }}Shorter than the input, or the same length when nothing repeated.
{{ 5.resultCount }} returns the number of items in {{ 5.results }}.
Examples to copy
One email per person
Two exports combined into one list, some people in both.
- Compare by field:
email - Which copy to keep: Last
- Ignore capitalisation: on
Last is the choice when the second export is the more recent one and its values should win. Ignoring capitalisation catches [email protected] and [email protected] as one person.
Unique tags
A list of plain text values with repeats.
- Compare by field: leave empty, so the value itself is compared
- Which copy to keep: First
Limits
Remove duplicates works on lists of up to 100,000 items. A longer list is refused with a message naming both numbers.
- Run
- Pass a list with repeated values, set Compare by field (or leave empty for plain values), and click Run step in Test & Debug
- You should see
- The Output tab shows results with duplicate entries removed and resultCount set to the surviving item count.