HashDup – Fast, Memory‑Safe Duplicate File Finder CLI

说明与用途

HashDup 是一个快速、内存安全的 CLI 重复文件查找器，采用两阶段算法：先按文件大小过滤，再使用流式 SHA‑256 哈希，保持内存使用恒定，避免 OOM 错误。

问题

Workstations accumulate duplicate files that waste SSD space; naive hashing is slow and can crash Node.js with large files.

结果

HashDup reduces duplicate detection time by filtering out unique‑size files first, then streams large files for hashing, keeping memory usage constant and preventing OOM errors. It reports duplicate sets and reclaimable space, enabling safe cleanup.

工作流

• Scan all target files with fs.promises.stat() to group by file size (Phase 1).
• Discard files with unique sizes immediately.
• For size‑matched files, stream each file in 64 KB chunks and compute SHA‑256 (Phase 2).
• Collect duplicate groups and calculate total space that can be reclaimed.
• Display results in a CLI table; dry‑run is default to avoid accidental deletions.
• Optionally delete duplicates after review.
• Use npx hashdup-cli scan <directory> to run.

证据

Phase 1: Size‑First Grouping (O(1) filter) – Discard all files with unique byte sizes without reading contents!

来源

Phase 2: Chunked Streaming SHA‑256 Hashing – Streams files in 64KB buffers (Flat O(1) memory consumption)

来源