Why I'm recommending dedupe

Описание и применение

dedupe позволяет эффективно удалять дубликаты поставщиков, используя блокировку, обученные веса полей и транзитивное кластеризование, что приводит к точному подсчёту уникальных записей.

Задача

Managing 1.2M supplier records across three systems produced 340k “unique” suppliers via exact‑match joins, while the true unique count was ~190k due to variations in company names, typos, and trailing spaces.

Результат

dedupe reduced duplicate supplier records from 340k to ~190k, achieving 5000 records/sec throughput on 1M datasets and linear scalability with parallel partitioning.

Рабочий процесс

• 1. Label ~50 pairs of records that are duplicates or unique.
• 2. Run dedupe’s blocking stage (e.g., Soundex, n‑gram signatures) to limit candidate pairs.
• 3. Apply probabilistic matching with learned field weights to score candidate pairs.
• 4. Perform transitive clustering (GLM algorithm) to merge all related duplicates into single entities.
• 5. Export the cleaned, deduplicated dataset for downstream use.

Доказательства

We had 1.2M supplier records across three systems. Same company written four ways… Exact‑match joins returned 340k “unique” suppliers. The real number was closer to 190k.

Источник

dedupe is what we ran instead of writing another regex pile. It learns field weights from labeled pairs rather than asking you to hand‑tune a similarity threshold.

Источник