Remove duplicate lines online, free and private
To remove duplicate lines from text for free and without uploading it, paste your list into the Input Text box, or click Choose File to open a .txt, .csv or .log file. The tool keeps one copy of every line and shows the result under Unique Lines as you type. Click Copy or Download .txt to save it. The work happens in your browser, and the Network tab in DevTools shows that pasting text sends nothing.
A duplicate line remover compares whole lines and drops the repeats. By default it keeps the first copy of each line in its original position, and it ignores spaces and tabs at the start and end of a line when it compares. You can switch to case-insensitive matching, keep the last copy instead, sort the result A-Z, or choose Show Duplicates Only to see which lines repeat before you delete anything.
When the list is clean, put it in order with the sort lines alphabetically tool, drop empty rows with remove empty lines, or compare two versions of a list in the text diff tool.
Table of Use: settings and what you get
Each row shows what you paste in, the setting you change from its default, and the result. āµ marks a line break and ā£ a space.
| Task | Input | Setting | Result |
|---|---|---|---|
| Clean a plain list | pearāµappleāµpearāµfig | Defaults | pearāµappleāµfig |
| Merge a list that mixes capitals | ParisāµromeāµparisāµRome | Case sensitive unticked | Parisāµrome |
| Ignore stray spaces | appleā£āµapple | Ignore leading/trailing spaces ticked (the default) | appleā£ (first copy, kept as written) |
| Keep the newest entry | xāµyāµxāµzāµy | Keep Last Occurrence | xāµzāµy |
| See which lines repeat | xāµyāµxāµzāµy | Show Duplicates Only | xāµxāµyāµy |
| Dedupe and sort in one step | item 10āµitem 2āµitem 10āµitem 1 | Preserve order unticked | item 1āµitem 2āµitem 10 |
| Drop blank lines as well | pearāµappleāµpearāµāµappleāµāµfig | Remove blank lines ticked | pearāµappleāµfig (without it, one blank line stays) |
Features
- Four switches. Case sensitive (on by default), Ignore leading/trailing spaces (on), Remove blank lines (off) and Preserve order (on).
- Keep First or Last Occurrence. The line that survives stays at the position of the copy you kept.
- Show Duplicates Only. Lists every copy of each repeated line, grouped together in the order each line first appears.
- Statistics. Total lines, Distinct lines, Lines with duplicates and, in the unique view, Lines removed, updated as you type.
- Your lines are not rewritten. Case and spaces are ignored only for comparing; the line you keep is output exactly as written.
- Accents match. Lines are compared after Unicode normalisation, so an Ć© typed as one character equals an e followed by a combining accent.
- Line endings kept. Unix, Windows and old Mac line breaks are all read, the output reuses the line ending of your text, and a final line break is not counted as an extra blank line.
- File in, file out. Open .txt, .md, .json, .xml, .csv, .tsv or .log files; download deduplicated-lines.txt (or duplicate-lines.txt in the duplicates view).
- Use Result and Reset. Use Result feeds the output back in for a second pass with different settings; Reset clears the text and restores every default.
Check the outcome with the line counter, label it with add line numbers, or fix a recurring typo across the list with find and replace.
How to Use the Duplicate Line Remover
- Paste your list into Input Text, one item per line, or click Choose File to open a text file.
- Under View Mode, leave Show Unique Lines selected to delete repeats, or pick Show Duplicates Only to inspect them.
- Under Keep Mode, choose Keep First Occurrence or Keep Last Occurrence.
- Tick or untick Case sensitive, Ignore leading/trailing spaces and Remove blank lines. Untick Preserve order to sort the result A-Z.
- Read the Statistics, then check the lines under Unique Lines.
- Click Copy or Download .txt. Click Use Result to run another pass, or Reset to start over.
Who Can Use It
Marketing and sales teams merge mailing lists and then check the survivors with the email validator. SEO writers tidy keyword and URL lists, and can standardise capitals afterwards with the text case converter.
Developers and system administrators clean log excerpts, hosts files and dependency lists. Analysts paste a single spreadsheet column here before turning the rows into data with the CSV to JSON converter. Teachers and students de-duplicate vocabulary and reading lists, then measure them with the word counter or character counter.
FAQs
How do I remove duplicate lines from text for free?
Paste your text into the Input Text box with one item per line. The result appears straight away under Unique Lines, with the first copy of every line kept in its original position. Click Copy to put it on the clipboard or Download .txt to save it. There is no sign-up, no limit on uses and no upload, because the work is done in your browser.
How do I remove duplicate lines from a text file?
Click Choose File under "Or open a text file" and pick a .txt, .md, .json, .xml, .csv, .tsv or .log file. Your browser reads it locally as text and fills the Input Text box. Adjust the settings if needed, then click Download .txt to save the result as deduplicated-lines.txt. The original file on your device is not changed.
How do I remove duplicate lines and ignore case?
Untick Case sensitive. Lines such as "Paris" and "paris" are then treated as the same line, and only one is kept. The kept line is written exactly as you typed it, so with Keep First Occurrence you get the capitalisation of the earliest copy. Keep Last Occurrence returns the capitalisation of the final copy instead.
Why are lines that look identical not removed?
They differ in something you cannot see. Capitalisation is the usual cause, so untick Case sensitive. Spaces or tabs at the start or end are ignored while Ignore leading/trailing spaces is ticked. A tab instead of a space inside the line, a non-breaking space or a zero-width character in the middle still makes two lines different, because only the edges are trimmed.
How do I find duplicate lines instead of deleting them?
Click Show Duplicates Only under View Mode. The output then lists every copy of each line that appears more than once, with the copies grouped together in the order each line first appears, and leaves out lines that appear only once. The Statistics panel shows how many distinct lines have duplicates, so you can see the scale before you decide what to delete.
What is the difference between Keep First Occurrence and Keep Last Occurrence?
Keep First Occurrence keeps the earliest copy of a repeated line and removes the later ones. Keep Last Occurrence keeps only the final copy. Either way the surviving line stays at the position of the copy that was kept, so with a log or export where newer entries come later, Keep Last Occurrence leaves you with the most recent version of each line.
Does removing duplicates change the order of my lines?
Not by default. Preserve order is ticked, so the lines that remain stay in their original sequence. Untick it to sort the unique lines A-Z using your browser's language rules, with numbers in natural order, so "item 2" comes before "item 10". Blank lines are treated like any other line unless you tick Remove blank lines.
Is it safe to remove duplicates from a customer or email list online?
Yes. The text is processed by JavaScript in your browser, and a file you open is read locally, so nothing is uploaded to ZipConvert or anyone else. Open the Network tab in DevTools while you paste a list and you will see no request. Nothing is saved either: closing or refreshing the tab clears the text.