I benchmarked 4 vector DBs with identical vectors — retrieval was identical, scores were not

Описание и применение

Автор сравнил четыре векторных базы данных, обнаружил несоответствие шкалы оценок и исправил ошибки, обеспечив одинаковый результат поиска и корректную нормализацию оценок.

Задача

Ensuring backend portability for vector databases requires that retrieval quality and score normalization remain consistent across different storage engines.

Результат

All four backends achieved perfect overlap@10, but score normalization bugs were discovered and fixed, reducing mean absolute score deviation from 0.17561 to 0.00003.

Рабочий процесс

• 1. Prepare 296 SQuAD paragraphs and 60 questions. 2. Generate identical vectors for pgvector, FAISS, Qdrant, and Weaviate. 3. Run benchmark with ./benchmark/run.sh to start all backends. 4. Compute ground truth via exact float64 scan. 5. Query each backend for top‑10 results. 6. Calculate overlap@10 and mean absolute score deviation. 7. Identify and fix score scale mismatch and three edge‑case bugs. 8. Verify results in benchmark/RESULTS.md and results.csv.

Доказательства

overlap@10 = 1.000 on all four

Источник

Found 5 bugs: score scale mismatch + 3 edge cases

Источник

Now deviation: 0.17561 -> 0.00003

Источник