Remove Duplicate Lines from Text
Paste your text, click remove — done. No sign-up, no API, nothing leaves your browser. Works on lists, emails, CSV data, code, and more.
A duplicate line is any line of text that appears more than once in a document. A remove duplicate lines tool scans your input line by line, tracks which lines it has already seen, and discards every subsequent repeat — keeping exactly one copy of each unique line while preserving the original order.
How to Remove Duplicate Lines — Step by Step
This tool works entirely inside your browser. Nothing is uploaded, saved, or transmitted. Here is exactly what to do:
.txt file. That's it.
Every Option Explained
Each toggle is designed for a real-world scenario. Here is what each one does and when to use it:
Trim whitespace
Strips invisible leading and trailing spaces before comparing lines. Without this, "apple" and "apple " (with a trailing space) would be treated as different. Recommended for almost all use cases.
Case-insensitive
Treats Apple, APPLE, and apple as the same line. Essential for email address lists and keyword sets where capitalization varies by source. The tool keeps the original capitalization of the first occurrence.
Remove blank lines
Deletes completely empty lines from the output. Particularly useful after processing CSV exports or data copied from spreadsheets, which often include stray blank rows.
Sort A–Z / Z–A
Alphabetically sorts the deduplicated output. A–Z is ascending (Aaron before Zara), Z–A is descending. These two options are mutually exclusive — selecting one automatically deselects the other.
Reverse order
Flips the sequence after deduplication — the last line becomes the first. Useful for server log files or timestamped lists where the most recent entry should appear at the top.
Keep last occurrence
By default, the first time a line appears is kept. Enable this to keep the most recent occurrence instead. Useful when lines represent records that were updated — the last version is the correct one.
Number lines
Prefixes each output line with a sequential number: 1. line one, 2. line two, etc. Handy for producing numbered lists, or for referencing specific lines in a review or document.
Show duplicates panel
Opens a panel below the editors listing every line that was found more than once, with a count of how many times it appeared. Sorted by frequency — the most duplicated lines appear first.
Toolriz vs Other Methods: Side-by-Side Comparison
There are several ways to remove duplicate lines from text. Here is how this tool compares to the most common alternatives that professionals in the US use daily:
| Method | Preserves order | Case control | No install | Works on mobile | Dup report | Privacy |
|---|---|---|---|---|---|---|
| Toolriz (this tool) | ✓ | ✓ | ✓ | ✓ | ✓ | 100% local |
| Excel / Google Sheets | ✓ | Limited | Needs app | Partial | ✗ | Cloud sync |
sort -u (Linux/Mac) |
✗ (sorts) | Flag only | Needs terminal | ✗ | ✗ | Local |
| Notepad++ TextFX | ✗ (sorts) | ✓ | Needs install | ✗ | ✗ | Local |
| Python script | ✓ | ✓ | Needs Python | ✗ | Possible | Local |
| Other online tools | Usually | Varies | ✓ | Varies | Rarely | Server upload |
Who Uses This Tool and Why
These are the real-world scenarios where professionals reach for a duplicate line remover most often:
Email list hygiene
Marketing and operations teams accumulate duplicate subscriber addresses across campaigns, form submissions, and CRM exports. Paste your list here with case-insensitive mode on, and get a clean unique set in seconds — no VLOOKUP, no formulas, no spreadsheet headaches.
SEO keyword deduplication
SEO professionals merging keyword lists from Google Search Console, Ahrefs, Semrush, and keyword planners end up with significant overlap. This tool strips the repeats before import into a tracker — saving filtering time downstream.
Code & config cleanup
Developers working with large import blocks, package.json dependency lists, .env files, or YAML configs can quickly strip accidentally doubled entries without writing a one-off script or using grep.
Database & CRM data prep
Before loading records into PostgreSQL, MySQL, Airtable, or any CRM, duplicate rows cause constraint violations and bloated tables. Run your CSV column through here first to catch repeats before they break an import.
Research & writing
Journalists, researchers, and content writers assembling reference lists, source URLs, citations, or bibliographies from multiple sources inevitably end up with repeated entries. One paste and click clears them all.
Security & access control
System administrators auditing IP allowlists, firewall rules, user permission sets, or SSH authorized_keys files use this tool to quickly find and remove redundant entries that bloat configs and obscure the actual policy.
How Duplicate Detection Works Under the Hood
Understanding the algorithm helps you choose the right options for your data. The tool uses a hash map (JavaScript Map object) for O(n) time complexity — it processes each line exactly once, regardless of how many lines you have.
The core algorithm
For each line in your input, the tool computes a lookup key (after applying trim and case-folding if enabled) and checks if that key already exists in the map. If it does, the line is a duplicate and is counted but not added to the output. If it doesn't, it's added to the map and kept. This approach is significantly faster than sorting-based deduplication (O(n log n)) and preserves original order, which sort-based methods cannot.
const seen = new Map();
const result = [];
lines.forEach(line => {
let key = trimWS ? line.trim() : line;
if (caseInsensitive) key = key.toLowerCase();
if (!seen.has(key)) {
seen.set(key, true);
result.push(line); // keep original formatting
}
// else: duplicate — skip
});Why order is preserved
Many tools (including the Unix sort -u command) deduplicate by first sorting all lines, which inevitably changes their order. This tool's hash-map approach processes lines sequentially, so the output order always matches the input order — only duplicates are removed in place.
Case-insensitive matching detail
When case-insensitive mode is on, the comparison key is lowercased, but the stored line retains its original formatting. So if your input has Apple, APPLE, and apple, the output keeps Apple (the first occurrence, original case) and discards the other two.
Map uses hash-based lookups with average O(1) per operation. Processing 1,000,000 lines typically takes under 500ms on a modern laptop. Memory usage scales linearly with the number of unique lines — not total lines.Frequently Asked Questions
Answers written to directly address common searches. If your question isn't here, the answer is almost certainly in one of the sections above.
sort -u input.txt > output.txt — removes duplicates but also sorts the output alphabetically. (2) awk '!seen[$0]++' input.txt > output.txt — removes duplicates while preserving the original line order. This tool replicates option 2 in a browser GUI, with additional options like case-insensitive matching.lines = list(dict.fromkeys(open('input.txt').read().splitlines())). For case-insensitive: use a seen set with line.lower() as the key while appending the original line. If you'd rather not write code, this tool does the same thing visually.