duplicut

Duplicut :scissors:

Quickly dedupe massive wordlists, without changing the order

---

:book: Overview

Nowadays, password wordlist creation usually implies concatenating multiple data sources.

Ideally, most probable passwords should stand at start of the wordlist, so most common passwords are cracked instantly.

With existing dedupe tools you are forced to choose if you prefer to preserve the order OR handle massive wordlists.

Unfortunately, wordlist creation requires both

So i wrote duplicut in highly optimized C to address this very specific need :nerd_face: :computer:

---

:bulb: Quick start

git clone https://github.com/nil0x42/duplicut  # download ...
cd duplicut/ && make                           # compile ...
./duplicut wordlist.txt > clean-wordlist.txt   # dedupe !!!

:wrench: Options

-o, --outfile <FILE>       Write result to <FILE> (default stdout)
-t, --threads <NUM>        Worker threads (default online CPUs)
-m, --memlimit <VALUE>     Memory budget for sizing (default available)
-l, --line-max-size <NUM>  Max line size in bytes (default 1024, max 4095)
-p, --printable            Keep only printable ASCII lines
-c, --lowercase            Convert wordlist to lowercase
-C, --uppercase            Convert wordlist to uppercase
-D, --dupfile <FILE>       Write dupes to <FILE> (slows down duplicut)
-h, --help                 Display this help and exit
-v, --version              Output version information and exit

Without -o, duplicut writes to stdout. You can redirect the result, append it to an existing file, or pipe it to another command:

./duplicut wordlist.txt > clean-wordlist.txt
./duplicut wordlist.txt >> combined-wordlist.txt
cat wordlist.txt | ./duplicut | another-command

• Features:

  • Handle massive wordlists, even those whose size exceeds available RAM
  • Filter lines by max byte length (--line-max-size option)
  • Filter-out lines with non-printable ASCII chars (--printable option)
  • Save found duplicates to another file (--dupfile option)
  • Ignore case with --lowercase or --uppercase options
  • Displays a nice progress bar with ETA

• Implementation:

  • Written in pure C code, designed to be fast
  • Compressed hashmap items on 64 bit platforms
  • Multithreading support

• Limitations:

  • --line-max-size can’t exceed 4095

:book: Technical Details

:small_orange_diamond: 1- Memory optimized:

Each hashmap entry packs a line address and its length into a 64-bit word using a tagged pointer: 12 bits for the byte length and 52 bits for the address. This assumes the upper 12 address bits are zero.

:small_orange_diamond: 2- Massive file handling:

The working file is memory-mapped and split into chunks. The target chunk size is estimated from the hash table capacity; chunk boundaries are extended to the next newline.

Each chunk is indexed in the hashmap and deduped. Subsequent chunks are then checked against that table.

For n chunks, this schedules n(n+1)/2 chunk-processing tasks. This task count is not an execution-time bound:

:bulb: Troubleshooting

If you find a bug, or something doesn’t work as expected, please compile duplicut in debug mode and post an issue with attached output:

# debug level can be from 1 to 4
make debug level=1
./duplicut [OPTIONS] 2>&1 | tee /tmp/duplicut-debug.log