The 480 Outputs That Never Became Actions

Description and use

During multi‑agent AI experiments, the system can run correctly yet fail to produce observations that answer the research question. In MOSAIC, all 480 generations from the qwen2.5:7b population contained a write_artifact prefix that violated the JSON output contract, leading to zero usable observations. The experiment stopped automatically, but the research question remained unanswered.

Problem

When running multi‑agent AI experiments, the system can execute as designed yet fail to produce observations that answer the research question.

Outcome

The S2 run failed to produce usable observations due to a repeated write_artifact prefix, but the system correctly detected the failure, stopped the experiment, and provided evidence of the instrument’s behavior under failure conditions.

Workflow

• 1. Record model generations with full request context.
• 2. Replay recorded generations to validate execution pipeline.
• 3. Detect output format violations via strict parser.
• 4. Classify unusable pairs and trigger registered stopping rule.
• 5. Investigate generation records to identify common prefix issue.
• 6. Document failure mode and instrument behavior.
• 7. Decide on next steps through research governance.

Evidence

Replaying an LLM execution is not the same as independently reproducing the model's behaviour.

Source

All 480 distinct generations exhibited the same relevant output pattern: a bare write_artifact prefix followed by a JSON object.

Source

The system did not need someone to intervene manually once the registered condition was met.

Source