I Sell Memory APIs. I'm Also Building the Benchmark. Here's How I'm Trying Not to Rig It.

Описание и применение

Создан открытый и прозрачный бенчмарк для систем памяти, с четкими правилами подачи, публичными данными и справедливой оценкой скорости.

Memory benchmark results from vendors are inconsistent and untrustworthy, leading to lack of confidence in published numbers.

The benchmark framework is publicly available, with transparent rules and data, aiming to provide reliable memory system evaluations.

1. Define clear submission rules: all submissions undergo the same approval process and no self‑merging.
2. Publish per‑question records so anyone can recompute aggregate scores.
3. Set reader, judge, and prompts by version to prevent manipulation.
4. Require harnesses that are usable by customers to ensure reproducibility.
5. Release rules and code under Apache 2.0 to allow independent hosting.
6. Build a corpus of 1.9M tokens from 17 months of conversation, including 50 photos and multilingual sessions.
7. Design axes for evaluation: stale facts, contradictions, apparent contradictions, abstaining, and false‑memory probes.
8. Measure speed fairly by subtracting network floor, warming connections, and recording response size.
9. Score speed per question on a log scale, capped between -1 and +1, and average results.
10. Publish benchmark repository on GitHub (github.com/wontopos/glasshouse).

Wontopos hosts it and keeps it running. That is the whole role.

Publish the per-question record. Anyone can recompute the number from it.

Speed is reported next to accuracy, not folded into it silently, and the weight is stated.