The 480 Outputs That Never Became Actions

说明与用途

在多智能体 AI 实验中，系统可能按预期运行，但无法生成满足研究问题的观察结果。在 MOSAIC 中，qwen2.5:7b 人口的所有 480 次生成都包含 write_artifact 前缀，违反了 JSON 输出合同，导致零可用观察。实验自动停止，但研究问题仍未得到解答。

问题

When running multi‑agent AI experiments, the system can execute as designed yet fail to produce observations that answer the research question.

结果

The S2 run failed to produce usable observations due to a repeated write_artifact prefix, but the system correctly detected the failure, stopped the experiment, and provided evidence of the instrument’s behavior under failure conditions.

工作流

• 1. Record model generations with full request context.
• 2. Replay recorded generations to validate execution pipeline.
• 3. Detect output format violations via strict parser.
• 4. Classify unusable pairs and trigger registered stopping rule.
• 5. Investigate generation records to identify common prefix issue.
• 6. Document failure mode and instrument behavior.
• 7. Decide on next steps through research governance.

证据

Replaying an LLM execution is not the same as independently reproducing the model's behaviour.

来源

All 480 distinct generations exhibited the same relevant output pattern: a bare write_artifact prefix followed by a JSON object.

来源

The system did not need someone to intervene manually once the registered condition was met.

来源