HashDup – Fast, Memory‑Safe Duplicate File Finder CLI

Описание и применение

HashDup – быстрый CLI‑инструмент для поиска дубликатов файлов, использующий двухступенчатый алгоритм: сначала группирует файлы по размеру, затем вычисляет SHA‑256 потоками, сохраняя память постоянной и предотвращая ошибки OOM.

Задача

Workstations accumulate duplicate files that waste SSD space; naive hashing is slow and can crash Node.js with large files.

Результат

HashDup reduces duplicate detection time by filtering out unique‑size files first, then streams large files for hashing, keeping memory usage constant and preventing OOM errors. It reports duplicate sets and reclaimable space, enabling safe cleanup.

Рабочий процесс

• Scan all target files with fs.promises.stat() to group by file size (Phase 1).
• Discard files with unique sizes immediately.
• For size‑matched files, stream each file in 64 KB chunks and compute SHA‑256 (Phase 2).
• Collect duplicate groups and calculate total space that can be reclaimed.
• Display results in a CLI table; dry‑run is default to avoid accidental deletions.
• Optionally delete duplicates after review.
• Use npx hashdup-cli scan <directory> to run.

Доказательства

Phase 1: Size‑First Grouping (O(1) filter) – Discard all files with unique byte sizes without reading contents!

Источник

Phase 2: Chunked Streaming SHA‑256 Hashing – Streams files in 64KB buffers (Flat O(1) memory consumption)

Источник