Remove Duplicates
The Remove duplicates step returns a list with repeats taken out. You say what makes two entries the same, and it keeps one of each.
Use it after combining sources that overlap. The same customer exported from two systems, an address on several orders, a lead that came in twice.
Deduplicating inside a single field? You can remove duplicates directly inside field tokens using Data Transformation with {{ 2.emails | remove_duplicates }} (or {{ 2.records | pluck:email | remove_duplicates }}). Use this dedicated Remove duplicates step when deduplicating complex multi-field records or processing large datasets (up to 100,000 items).
Setting it up
Choose the list
Pick it from the lists the field offers, which come from the steps before this one. Choose Enter it myself to type a reference such as {{ 3.results }} or paste a JSON list instead.
Say what makes two entries the same
For a list of plain values, the value itself. For a list of records, name the field, such as email or customer_id.
Choose which copy to keep
The first one seen, or the last.
Which copy you keep changes the data, not the order
The surviving items stay in the order they first appeared, whichever copy wins. Keeping the last one does not reverse anything.
What changes is which record survives, and with real data that matters. Say a customer appears twice, once with an old address and once with a new one. First keeps the old, Last keeps the new, assuming the newer record came later in the list.
For a list of plain values it makes no difference at all: the same value is the same value.
Capital letters count unless you say otherwise
Ignore capitalisation is off to begin with, so Active and active are two different values and both survive.
Turn it on when the same thing is spelled differently by two systems: one exports upper case, the other lower.
What it passes on
The list with repeats removed:
{{ 5.$full_result }}Shorter than the input, or the same length when nothing repeated.
The step also counts the items for you: {{ 5.resultCount }} is how many it passed on, and {{ 5.results }} is the list as the data panel offers it.
Examples to copy
One email per person
Two exports combined into one list, some people in both.
- Field:
email - Keep: Last
- Ignore capitalisation: on
Last is the choice when the second export is the more recent one and its values should win. Ignoring capitalisation catches [email protected] and [email protected] as one person.
Unique tags
A list of plain text values with repeats.
- Field: leave empty, so the value itself is compared
- Keep: First
Limits
Remove duplicates works on lists of up to 100,000 items. A longer list is refused with a message naming both numbers.