Finding the sentence that made an AI agent misbehave

Description and use

runtape lets you pinpoint the exact sentence causing an AI agent’s misbehavior and statistically verify fixes, ensuring reliable operation.

Problem

When an AI agent misbehaves, it is hard to identify which part of its context caused the wrong action, and a single rerun cannot confirm if a fix works.

Outcome

runtape successfully isolates the exact sentence causing misbehavior, applies a fix, and verifies it with statistical tests and automated tests, preventing recurrence.

Workflow

• Record agent runs to JSONL with runtape
• Run runtape why <decision> to identify the decision point
• Rerun the decision with unchanged context to get baseline frequency
• Iteratively remove each context piece (system prompt, messages, tool results) and rerun to see effect
• Use Fisher exact test to determine statistically significant causes
• Identify the minimal JSON item, paragraph, and sentence causing the action
• Apply fixes using runtape fix which tests four rule variations and source removal
• Verify fix by rerunning decision 10 times per variation
• Generate a pytest test file with --write-test to guard against regressions

Evidence

runtape records your agent's runs to a JSONL file. When one goes wrong, you point it at the bad decision: runtape why last tool:forward_email

Source

It then: 1) Reruns that one model call on the unchanged context, 2) Removes each piece of the context, 3) Narrows it down to the JSON item, paragraph and sentence.

Source

runtape fix tries four changes on the exact context that failed and reruns the decision 10 times with each.

Source