I Sell Memory APIs. I'm Also Building the Benchmark. Here's How I'm Trying Not to Rig It.

说明与用途

建立了一个透明、社区托管的内存系统基准测试，具有公开规则、共享数据和公平的速度测量。

问题

Memory benchmark results from vendors are inconsistent and untrustworthy, leading to lack of confidence in published numbers.

结果

The benchmark framework is publicly available, with transparent rules and data, aiming to provide reliable memory system evaluations.

工作流

• 1. Define clear submission rules: all submissions undergo the same approval process and no self‑merging.
• 2. Publish per‑question records so anyone can recompute aggregate scores.
• 3. Set reader, judge, and prompts by version to prevent manipulation.
• 4. Require harnesses that are usable by customers to ensure reproducibility.
• 5. Release rules and code under Apache 2.0 to allow independent hosting.
• 6. Build a corpus of 1.9M tokens from 17 months of conversation, including 50 photos and multilingual sessions.
• 7. Design axes for evaluation: stale facts, contradictions, apparent contradictions, abstaining, and false‑memory probes.
• 8. Measure speed fairly by subtracting network floor, warming connections, and recording response size.
• 9. Score speed per question on a log scale, capped between -1 and +1, and average results.
• 10. Publish benchmark repository on GitHub (github.com/wontopos/glasshouse).

证据

Wontopos hosts it and keeps it running. That is the whole role.

来源

Publish the per-question record. Anyone can recompute the number from it.

来源

Speed is reported next to accuracy, not folded into it silently, and the weight is stated.

来源