Neura Market
nil0x42/duplicut logo

nil0x42/duplicut

Free

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Visit Website GitHub (38 stars)

About nil0x42/duplicut

Duplicut is an open-source command-line tool written in C that removes duplicate lines from massive wordlists while preserving the original order. Designed specifically for dictionary-based password cracking, it solves the problem of deduplicating files larger than available RAM without resorting to sorting (which reorders lines). The tool splits the wordlist into virtual chunks, uses a memory-optimized 64-bit compressed hashmap, and supports multithreading for performance. Additional features include filtering by line length, filtering non-printable characters, case conversion (lowercase/uppercase), saving duplicates to a separate file, and a progress bar with ETA.

Key Features

Handle massive wordlists that exceed available RAM
Preserve original order of wordlist
Multithreading support
Filter lines by maximum length (--line-max-size)
Filter out lines with non-printable ASCII characters (--printable)
Convert wordlist to lowercase or uppercase (--lowercase, --uppercase)
Save duplicate lines to a separate file (--dupfile)
Display progress bar with estimated time
Memory-optimized 64-bit compressed hashmap
Written in pure C for performance

Pros & Cons

Pros
  • Preserves original wordlist order (unlike sort -u)
  • Handles wordlists larger than available RAM via virtual chunking
  • Multithreaded for fast deduplication
  • Memory efficient with compressed hashmap on 64-bit
  • Open source with GPL-3.0 license
  • Simple command-line interface with useful options
  • Written in C for minimal overhead
Cons
  • Maximum line length limited to 4095 characters
  • Command-line only, no graphical interface
  • Primarily targeted at password cracking domain, not general-purpose deduplication
  • Requires compilation from source (C compiler needed)

Best For

Dictionary-based password cracking preparationConcatenating and deduplicating multiple wordlist sourcesRemoving duplicates from large text files while preserving line orderPreparing wordlists for hashcat or similar cracking tools

FAQ

What is duplicut?
Duplicut is a tool to remove duplicate lines from massive wordlists without sorting, preserving the original order. It is written in C and designed for password cracking preparation.
How does duplicut handle files larger than RAM?
It splits the file into virtual chunks that fit into memory, processes each chunk into a hashmap, and dedupes against subsequent chunks.
Is duplicut open source?
Yes, it is licensed under GPL-3.0.
What are the system requirements?
A C compiler (e.g., gcc) and a Unix-like operating system (Linux, macOS, etc.).
Can I limit memory usage?
Yes, with the --memlimit option.
Can I save the duplicates to a separate file?
Yes, with the --dupfile option.
How is duplicut different from sort -u?
sort -u sorts the file (changing order) and can require more memory. Duplicut preserves the original order and handles files larger than RAM.