I Built a Crash-Safe Key-Value Store Without sqlite3 — Then I Killed It

Description and use

Anvil is a crash‑safe, embedded key‑value store built entirely with Python’s standard library, showing how to manage an append‑only log, in‑memory index, checksums, and atomic compaction without external dependencies.

Problem

Build a crash‑safe, embedded key‑value store with no third‑party runtime dependencies, exposing all persistence guarantees in code.

Outcome

Anvil demonstrates that a fully crash‑safe key‑value store can be built with only the Python standard library, exposing all persistence guarantees and passing rigorous crash‑harness tests.

Workflow

• Design append‑only log and binary record format using struct and zlib.crc32.
• Implement in‑memory index mapping keys to log offsets.
• Add single‑writer locking with fcntl/msvcrt.
• Implement crash harness: start writer, perform writes, kill with SIGKILL, reopen store and verify all acknowledged writes are intact.
• Implement compaction: write temporary compacted file, fsync, atomic os.replace, fsync directory.
• Add regression tests for corruption scenarios (e.g., altered length fields).
• Benchmark write/read throughput and database size.
• Publish reproducible single‑file builds via zipfile tooling.

Evidence

The write‑path crash harness passed 8/8 real SIGKILL iterations.

Source

The benchmark reported 6,899.5 ops/sec write throughput and 200,015.8 ops/sec read throughput.

Source