ToolsHub4u

Remove duplicate lines, words or list items

Paste a list and the repeats disappear in one pass. Choose whether an item is a line, a word, a sentence or a comma-separated value, then decide whether case and stray spaces should count.

  • Lines, words, sentences or CSV
  • Case-insensitive option
  • Keeps first occurrence
Ready Text & Writing
Which copy survives?The first occurrence is kept by default, so the order of your list is preserved. Sorting is optional and changes that order.
1Enter
2Adjust
3Copy result

Remove Duplicate Lines

1

List and matching rules

Duplicates are removed from the whole text, not just adjacent lines.

Treat each item as
Matching rules
No text yet Live
2

Unique list

Items kept

5

Removed 2 duplicates

Unique lines

Original items
7
Duplicates removed
2
Result characters
30

Matching is exact apart from the options you selected. Apple and apple are different items.

Paste a list and the repeats disappear. The default treats each line as one item, trims the spaces around it and drops blank lines, which is what you want for a column of URLs, emails or product codes copied out of a spreadsheet.

A worked example with the default options plus ignore case:

Worked exampleInput: apple / banana / Apple / banana / (blank) / cherry
Output: apple / banana / cherry
Result: 3 items kept, 2 duplicates removed

"Apple" disappeared because case was ignored and the first spelling — lowercase "apple" — is the one kept. With ignore case off, the same list keeps both spellings and only the repeated "banana" goes.

Four ways to define an item

What each mode treats as one item and how the output is joined
ModeAn item isOutput joins with
LinesEverything between two line breaksA new line
WordsEverything between two spacesA single space
SentencesText up to the next full stop, question or exclamation markA single space
Comma valuesText between commas or line breaksA comma and a space

Comma values is the mode people miss. It splits on commas and on newlines at the same time, so a list pasted from a single cell and a list pasted from a column are handled by the same click. The output is rebuilt as apple, banana, cherry, ready to paste back into a spreadsheet cell.

Words mode is the lossy one. It rebuilds the text with single spaces, so line breaks, indentation and the original spacing are all gone. Use it to deduplicate a vocabulary list, not to clean a paragraph.

The four options, and the order they run

  • Trim spaces (on by default) removes whitespace from both ends of every item. Without it,  apple and apple are different items.
  • Remove empty (on by default) drops items that are empty after trimming, which is what blank lines between paragraphs become.
  • Ignore case (off by default) lowercases each item before comparing, so Apple and apple collide. The spelling that survives is the first one in the list.
  • Sort A–Z (off by default) reorders what is left, using the browser's locale-aware comparison.

Order matters, and the tool applies them in a fixed sequence: trim, then drop empties, then case-fold for comparison, then deduplicate, then sort. Sorting last is why the kept copy of a duplicate is always the first occurrence in your original order — turning sort on does not change which spelling wins, only where it ends up.

One counting quirk worth knowing. The "original items" figure is the kept items plus the duplicates removed, so items dropped as empty are not counted at all. In the worked example above, six lines went in and the stat reads 5, because the blank line vanished before the tally.

Sentences mode is the weakest one

Sentence splitting depends on punctuation, and this site's splitter masks abbreviations by replacing their full stop before splitting. The mask is broad: Sat, Sun, Mon, Wed, No, Fig and St are all on the abbreviation list, so any sentence ending in the word "sat" or "sun" is glued to the next one.

Type the cat sat. the cat sat. the dog ran. and the tool reports no duplicates, because it sees one long sentence rather than three. The fault is in the splitting, not the deduplication. If your text has sentences ending in those words, use lines mode with one sentence per line instead.

Sentences mode also joins its output with spaces, so paragraph breaks are lost. That is fine for a deduplicated list of one-line sentences, and wrong for prose you intend to keep.

What counts as a duplicate

Matching is exact after the options are applied. There is no fuzzy matching, no similarity score and no normalisation of punctuation or Unicode. New York and new york are duplicates only with ignore case on. New York and New York with a non-breaking space are never duplicates, because the trim option removes ordinary whitespace at the ends of an item, not a space in the middle.

Trailing commas and other punctuation stay attached in words mode, so apple, and apple are two different items. That is often the reason a "dedupe" appears to do nothing on a list copied from prose.

Limits and habits

  • No counts or reports. You get items kept, duplicates removed and the character count of the result. There is no table of which item appeared how many times.
  • No fuzzy or near-duplicate removal. For that, compare the two lines with the similarity checker first.
  • No undo. Applying the result replaces your input, so copy the original if the order matters.
  • Text stays in the tab. Nothing is uploaded and a reload clears the box.

For a mailing list, the safe routine is: trim and remove empty on, ignore case on, sort on, then read the first and last few lines. Sorting brings variants of the same entry next to each other, which is how you spot the ones exact matching cannot merge — Acme Ltd next to Acme Limited.

Questions

Common questions

Short answers to what people ask before using this tool.

Does it remove duplicates or all copies?

It removes the repeats and keeps the first copy. Nothing is deleted from the input until you apply the result, and no item is removed entirely unless you delete it yourself.

Which copy is kept when case is ignored?

The first one in your list. With ignore case on, "apple" then "Apple" keeps "apple"; reverse the order and "Apple" survives. Sorting does not change which copy is kept, because sorting runs after deduplication.

Why did my blank lines disappear without being counted?

Because empty items are dropped before the tally is taken. The "original items" figure counts kept items plus duplicates removed, so blank lines and whitespace-only lines are not in it.

Can it ignore punctuation or spacing inside an item?

No. Trim only removes whitespace at the start and end of an item. "apple," and "apple" are different items in words mode, and a non-breaking space inside a phrase will stop two otherwise identical entries matching.

Why does sentences mode say there are no duplicates?

Probably because the sentences were not split. The splitter masks short words such as "sat", "sun" and "mon" as abbreviations, so a sentence ending with one runs into the next and the repeated text never becomes two separate items.

Does it work on comma-separated lists?

Yes — choose comma values. It splits on commas and on line breaks, so a row pasted from a spreadsheet and a column pasted from a file both work. The output is rebuilt with a comma and a space between items.

Is there a limit on list size?

There is no stated limit, and everything runs in the tab rather than on a server. In testing, a 20,000-line list with 15,000 unique entries was deduplicated in well under a second, though the browser redraws the result after every keystroke, so a very large paste can feel slow. Split the file if that happens.

Is my list uploaded anywhere?

No. Processing happens locally in the tab. Reloading the page clears the box, so copy the result before you close it. Avoid pasting anything confidential into any browser tool you do not control.