Building a Vector Similarity Detector: How One SQL Query Over 2.9M Charity Pairs Reveals the Gap Between Meaning and Spelling

说明与用途

文章展示了在 SQL 内部计算嵌入的余弦相似度，揭示了慈善机构名称在拼写与语义之间的差距，从而提升匹配准确性。

问题

Traditional fuzzy matching fails to capture semantic similarity between charity names, leading to false positives and missed matches.

结果

Embedding‑based similarity computed in SQL revealed a significant gap between spelling and meaning, improving matching accuracy for charity names.

工作流

• Collect charity names from multiple registries
• Generate all pairwise combinations (≈2.9M)
• Compute Jaro‑Winkler, Jaccard, and sentence‑transformer embeddings
• Store embeddings in a SQL table
• Create pairs table with lexical and token scores
• Insert embedding cosine similarities via SQL query
• Analyze score distributions to identify mismatch zones

证据

Approximately 12% of pairs with Jaro‑Winkler similarity above 0.85 ended up with embedding cosine similarity below 0.3.

来源