Why your agent's memory layer should make you prove it wrong

Описание и применение

Поток работы, позволяющий воспроизводить и проверять показатели памяти AI‑агентов, используя nautilus‑compass и VerifyPack.

Задача

AI‑agent memory layers often rely on implicit trust and opaque benchmarks, making it hard to verify their performance claims.

Результат

The workflow yields a memory layer that achieves higher precision@1 and precision@5 on LongMemEval‑S compared to mem0 2.0.19, while enabling full byte‑level reproducibility of benchmark claims.

Рабочий процесс

• Clone the nautilus‑compass repo
• Run the install script to set up the plugin
• Start the daemon to enable memory services
• Configure your MCP client to use the local or hosted gateway
• Run benchmark tests to compare against other memory layers
• Use VerifyPack commands to recompute and verify published numbers

Доказательства

P@1 0.890 vs 0.774, P@5 0.978 vs 0.916

Источник

Every number we publish, you should be able to recompute from bytes, without trusting us.

Источник

Two stdlib-only commands re-derive any number — no third-party dependencies, no "trust our notebook".

Источник