Give your AI agent a flight recorder

Description and use

Implementing a flight recorder for AI agents provides precise logging of checks, state transitions, and verdicts, uncovering real blocks and hidden errors invisible in git.

Problem

AI agent workflows lack traceability, making it hard to quantify failures, understand blocked work, and trust metrics derived from memory.

Outcome

The flight recorder captured ~200 events across ten tasks, revealing that only 2 real blocks occurred (not 64 failures), exposing blind spots in metrics and git visibility, and highlighting gaps such as unlogged pauses and missing rollback events.

Workflow

• 1. Add recorder hooks to enforcement points in the agent workflow.
• 2. Log each event as a JSON line with event type, phase, exit code, timestamp, and previous hash.
• 3. Run a checker script to validate chain integrity and timestamp monotonicity.
• 4. Analyze the log: compute naive metrics, map missing states, and uncover non-committed work.
• 5. Iterate to cover all task states and capture pauses or rollbacks.

Evidence

Agent work produces no trace by default, so every claim about it runs on memory, and memory is worse than wrong: it's confident.

Source

Our first reading of it was almost exactly wrong. The raw count said 86 checks ran and 64 of them exited non-zero — a 74 percent failure rate.

Source