I benchmarked 4 vector DBs with identical vectors — retrieval was identical, scores were not

说明与用途

作者对四个向量数据库进行了基准测试，发现评分尺度不匹配和三个边缘案例错误，修复后实现了检索质量和评分归一化的一致性。

问题

Ensuring backend portability for vector databases requires that retrieval quality and score normalization remain consistent across different storage engines.

结果

All four backends achieved perfect overlap@10, but score normalization bugs were discovered and fixed, reducing mean absolute score deviation from 0.17561 to 0.00003.

工作流

• 1. Prepare 296 SQuAD paragraphs and 60 questions. 2. Generate identical vectors for pgvector, FAISS, Qdrant, and Weaviate. 3. Run benchmark with ./benchmark/run.sh to start all backends. 4. Compute ground truth via exact float64 scan. 5. Query each backend for top‑10 results. 6. Calculate overlap@10 and mean absolute score deviation. 7. Identify and fix score scale mismatch and three edge‑case bugs. 8. Verify results in benchmark/RESULTS.md and results.csv.

证据

overlap@10 = 1.000 on all four

来源

Found 5 bugs: score scale mismatch + 3 edge cases

来源

Now deviation: 0.17561 -> 0.00003

来源