Three Raft bugs that only existed once I added log compaction

Описание и применение

Внедрение лог‑сжатия в реализацию Raft выявило три скрытых ошибки, которые обнаружил детерминированный симулятор, использующий виртуальное время и ин‑мемори сеть.

Задача

Raft implementations can hide subtle bugs behind log redundancy; log compaction exposes them.

Результат

Three bugs were uncovered: a late AppendEntries message misinterpreted as a conflict, a snapshotting edge case causing state divergence, and a log compaction race leading to inconsistent commit indices. Each bug was reproduced with a single seed and added to the test suite.

Рабочий процесс

• 1. Implement Raft core as pure synchronous state machine.
• 2. Replace nondeterministic components with interfaces (time, network, disk, scheduler).
• 3. Build deterministic simulator that runs the core on a single thread with a seeded random source.
• 4. Run extensive fuzz campaigns (10k seeds) and record failures.
• 5. Investigate failures to identify root causes and create regression tests.

Доказательства

Adding snapshotting exposed three real bugs.

Источник

The simulator runs the real consensus core, but not bbolt and not the node driver.

Источник

The event loop is a priority queue ordered by virtual time, and simultaneous events are routine — every node's first tick is at t=0.

Источник