Building a Vector Similarity Detector: How One SQL Query Over 2.9M Charity Pairs Reveals the Gap Between Meaning and Spelling

Description and use

The article demonstrates how computing cosine similarity of embeddings inside SQL uncovered a gap between spelling and meaning in charity names, improving matching accuracy.

Problem

Traditional fuzzy matching fails to capture semantic similarity between charity names, leading to false positives and missed matches.

Outcome

Embedding‑based similarity computed in SQL revealed a significant gap between spelling and meaning, improving matching accuracy for charity names.

Workflow

• Collect charity names from multiple registries
• Generate all pairwise combinations (≈2.9M)
• Compute Jaro‑Winkler, Jaccard, and sentence‑transformer embeddings
• Store embeddings in a SQL table
• Create pairs table with lexical and token scores
• Insert embedding cosine similarities via SQL query
• Analyze score distributions to identify mismatch zones

Evidence

Approximately 12% of pairs with Jaro‑Winkler similarity above 0.85 ended up with embedding cosine similarity below 0.3.

Source