I benchmarked 4 vector DBs with identical vectors — retrieval was identical, scores were not

Description and use

The author benchmarked four vector databases, identified a score‑scale mismatch and three edge‑case bugs, fixed them, and achieved consistent retrieval quality and score normalization.

Problem

Ensuring backend portability for vector databases requires that retrieval quality and score normalization remain consistent across different storage engines.

Outcome

All four backends achieved perfect overlap@10, but score normalization bugs were discovered and fixed, reducing mean absolute score deviation from 0.17561 to 0.00003.

Workflow

• 1. Prepare 296 SQuAD paragraphs and 60 questions. 2. Generate identical vectors for pgvector, FAISS, Qdrant, and Weaviate. 3. Run benchmark with ./benchmark/run.sh to start all backends. 4. Compute ground truth via exact float64 scan. 5. Query each backend for top‑10 results. 6. Calculate overlap@10 and mean absolute score deviation. 7. Identify and fix score scale mismatch and three edge‑case bugs. 8. Verify results in benchmark/RESULTS.md and results.csv.

Evidence

overlap@10 = 1.000 on all four

Source

Found 5 bugs: score scale mismatch + 3 edge cases

Source

Now deviation: 0.17561 -> 0.00003

Source