Building a Vector Similarity Detector: How One SQL Query Over 2.9M Charity Pairs Reveals the Gap Between Meaning and Spelling

Описание и применение

В статье показано, как вычисление косинусного сходства эмбеддингов внутри SQL выявило разрыв между орфографией и смыслом в названиях благотворительных организаций, улучшив точность сопоставления.

Задача

Traditional fuzzy matching fails to capture semantic similarity between charity names, leading to false positives and missed matches.

Результат

Embedding‑based similarity computed in SQL revealed a significant gap between spelling and meaning, improving matching accuracy for charity names.

Рабочий процесс

• Collect charity names from multiple registries
• Generate all pairwise combinations (≈2.9M)
• Compute Jaro‑Winkler, Jaccard, and sentence‑transformer embeddings
• Store embeddings in a SQL table
• Create pairs table with lexical and token scores
• Insert embedding cosine similarities via SQL query
• Analyze score distributions to identify mismatch zones

Доказательства

Approximately 12% of pairs with Jaro‑Winkler similarity above 0.85 ended up with embedding cosine similarity below 0.3.

Источник