The 480 Outputs That Never Became Actions

Описание и применение

При проведении экспериментов с многими ИИ‑агентами система может работать как задумано, но не генерировать наблюдения, отвечающие исследовательскому вопросу. В проекте MOSAIC обнаружили, что все 480 генераций модели qwen2.5:7b содержали префикс write_artifact, нарушающий контракт JSON‑вывода, из‑за чего наблюдения не использовались. Система корректно остановила эксперимент, но вопрос о влиянии модели остался без ответа.

Задача

When running multi‑agent AI experiments, the system can execute as designed yet fail to produce observations that answer the research question.

Результат

The S2 run failed to produce usable observations due to a repeated write_artifact prefix, but the system correctly detected the failure, stopped the experiment, and provided evidence of the instrument’s behavior under failure conditions.

Рабочий процесс

• 1. Record model generations with full request context.
• 2. Replay recorded generations to validate execution pipeline.
• 3. Detect output format violations via strict parser.
• 4. Classify unusable pairs and trigger registered stopping rule.
• 5. Investigate generation records to identify common prefix issue.
• 6. Document failure mode and instrument behavior.
• 7. Decide on next steps through research governance.

Доказательства

Replaying an LLM execution is not the same as independently reproducing the model's behaviour.

Источник

All 480 distinct generations exhibited the same relevant output pattern: a bare write_artifact prefix followed by a JSON object.

Источник

The system did not need someone to intervene manually once the registered condition was met.

Источник