Why I'm recommending dedupe

说明与用途

dedupe 通过阻塞、学习字段权重和传递聚类，有效去除供应商记录中的重复项，得到准确的唯一记录数。

问题

Managing 1.2M supplier records across three systems produced 340k “unique” suppliers via exact‑match joins, while the true unique count was ~190k due to variations in company names, typos, and trailing spaces.

结果

dedupe reduced duplicate supplier records from 340k to ~190k, achieving 5000 records/sec throughput on 1M datasets and linear scalability with parallel partitioning.

工作流

• 1. Label ~50 pairs of records that are duplicates or unique.
• 2. Run dedupe’s blocking stage (e.g., Soundex, n‑gram signatures) to limit candidate pairs.
• 3. Apply probabilistic matching with learned field weights to score candidate pairs.
• 4. Perform transitive clustering (GLM algorithm) to merge all related duplicates into single entities.
• 5. Export the cleaned, deduplicated dataset for downstream use.

证据

We had 1.2M supplier records across three systems. Same company written four ways… Exact‑match joins returned 340k “unique” suppliers. The real number was closer to 190k.

来源

dedupe is what we ran instead of writing another regex pile. It learns field weights from labeled pairs rather than asking you to hand‑tune a similarity threshold.

来源