Remove duplicate lines from a list
Keep only unique lines, optionally sorted alphabetically.
Calculator inputs
Results
Enter your values and press “Calculate” to see the result.
In short
- What it calculates
- Keep only unique lines, optionally sorted alphabetically.
- Formula used
The first occurrence of each line is kept- Example
- From the list apple, pear, apple, grape:
Remove duplicate lines
Lists of emails, product references, keywords or identifiers almost always arrive with repeats.
Paste the list and keep only what appears once.
How it works
Lines are scanned keeping the first occurrence of each value and discarding the rest. Optionally letter case is ignored and the result sorted.
Formula
The first occurrence of each line is kept
Removed = original − unique
Worked example
From the list apple, pear, apple, grape:
Original: 4
Unique: 3
Removed: 1
Explanation
Duplicates are almost never identical
A trailing space, a different capital, a mistyped accent or an invisible line break make two entries the eye reads as the same look different to a computer. Before removing duplicates it pays to normalise: trim spaces, unify case and decide whether accents should count.
Case: it depends on the data
In an email list, MARIA@EXAMPLE.COM and maria@example.com are the same person, because domains are case-insensitive. In a list of product codes or passwords, that distinction can be significant. There is no universal answer: it depends what the data represent.
Preserving order matters
When deduplicating a list ordered by priority, date or relevance, keeping the first occurrence of each value preserves that information. Sorting alphabetically to deduplicate and returning the sorted result destroys the original order, which was often the valuable part.
In databases, prevent rather than clean
A unique index stops duplicates entering in the first place. Cleaning afterwards is always costlier and riskier, especially if related records point at the rows about to be deleted.
Frequently asked questions
Is the original order preserved?
Yes, unless you turn on alphabetical sorting. Each value keeps the position of its first occurrence.
Is it case sensitive?
By default yes. With the ignore-case option, “London” and “london” count as the same line.
Why do duplicates still appear after cleaning?
Almost always trailing spaces, differing capitals or invisible characters. Normalising the text before comparing resolves most cases.
Is the original order preserved?
Yes. The first occurrence of each value is kept, which preserves the priority or date order of the original list.
Need to calculate something else?
These tools are often used alongside this calculator.
Change case
Turn text into uppercase, lowercase, title case, snake_case or camelCase.
Word count
Words, characters, sentences and paragraphs in any text.
Slug generator
Turn a title into a clean URL with no accents or odd characters.
Character count
Characters with and without spaces, lines and bytes in a text.
Reading time
How long a text takes to read, silently or aloud.