HashDup – Fast, Memory‑Safe Duplicate File Finder CLI

Description and use

HashDup is a fast, memory‑safe CLI duplicate file finder that uses a two‑phase algorithm: size‑based filtering followed by streaming SHA‑256 hashing, keeping memory usage constant and avoiding OOM errors.

Problem

Workstations accumulate duplicate files that waste SSD space; naive hashing is slow and can crash Node.js with large files.

Outcome

HashDup reduces duplicate detection time by filtering out unique‑size files first, then streams large files for hashing, keeping memory usage constant and preventing OOM errors. It reports duplicate sets and reclaimable space, enabling safe cleanup.

Workflow

• Scan all target files with fs.promises.stat() to group by file size (Phase 1).
• Discard files with unique sizes immediately.
• For size‑matched files, stream each file in 64 KB chunks and compute SHA‑256 (Phase 2).
• Collect duplicate groups and calculate total space that can be reclaimed.
• Display results in a CLI table; dry‑run is default to avoid accidental deletions.
• Optionally delete duplicates after review.
• Use npx hashdup-cli scan <directory> to run.

Evidence

Phase 1: Size‑First Grouping (O(1) filter) – Discard all files with unique byte sizes without reading contents!

Source

Phase 2: Chunked Streaming SHA‑256 Hashing – Streams files in 64KB buffers (Flat O(1) memory consumption)

Source