Why I'm recommending dedupe

Description and use

dedupe efficiently removes duplicate supplier records using blocking, learned field weights, and transitive clustering, yielding an accurate unique record count.

Problem

Managing 1.2M supplier records across three systems produced 340k “unique” suppliers via exact‑match joins, while the true unique count was ~190k due to variations in company names, typos, and trailing spaces.

Outcome

dedupe reduced duplicate supplier records from 340k to ~190k, achieving 5000 records/sec throughput on 1M datasets and linear scalability with parallel partitioning.

Workflow

• 1. Label ~50 pairs of records that are duplicates or unique.
• 2. Run dedupe’s blocking stage (e.g., Soundex, n‑gram signatures) to limit candidate pairs.
• 3. Apply probabilistic matching with learned field weights to score candidate pairs.
• 4. Perform transitive clustering (GLM algorithm) to merge all related duplicates into single entities.
• 5. Export the cleaned, deduplicated dataset for downstream use.

Evidence

We had 1.2M supplier records across three systems. Same company written four ways… Exact‑match joins returned 340k “unique” suppliers. The real number was closer to 190k.

Source

dedupe is what we ran instead of writing another regex pile. It learns field weights from labeled pairs rather than asking you to hand‑tune a similarity threshold.

Source