I Built a Crash-Safe Key-Value Store Without sqlite3 — Then I Killed It

Описание и применение

Anvil — это встроенный, crash‑safe key‑value store, построенный полностью на стандартной библиотеке Python, демонстрирующий, как можно управлять журналом, индексом, контрольными суммами и безопасной компакцией без сторонних зависимостей.

Задача

Build a crash‑safe, embedded key‑value store with no third‑party runtime dependencies, exposing all persistence guarantees in code.

Результат

Anvil demonstrates that a fully crash‑safe key‑value store can be built with only the Python standard library, exposing all persistence guarantees and passing rigorous crash‑harness tests.

Рабочий процесс

• Design append‑only log and binary record format using struct and zlib.crc32.
• Implement in‑memory index mapping keys to log offsets.
• Add single‑writer locking with fcntl/msvcrt.
• Implement crash harness: start writer, perform writes, kill with SIGKILL, reopen store and verify all acknowledged writes are intact.
• Implement compaction: write temporary compacted file, fsync, atomic os.replace, fsync directory.
• Add regression tests for corruption scenarios (e.g., altered length fields).
• Benchmark write/read throughput and database size.
• Publish reproducible single‑file builds via zipfile tooling.

Доказательства

The write‑path crash harness passed 8/8 real SIGKILL iterations.

Источник

The benchmark reported 6,899.5 ops/sec write throughput and 200,015.8 ops/sec read throughput.

Источник