Post Mortem: Terminal Bench Challenge - WASM Renderer

Описание и применение

Автоматизированные агенты увеличили локальные тесты с 64 до 1334, но рендерер остался неполноценно работающим, выявив пять ошибок в тестах и процессе разработки.

Задача

Agents’ local test passes rose from 64 to 1,334 while the renderer remained broken, revealing that many tests passed for the wrong reasons.

Результат

The renderer could render a basic triangle and pass many local tests, yet conformance remained low; five checks passed for the wrong reasons, exposing gaps in test design and workflow governance.

Рабочий процесс

• 1. Split work by role: architecture mapping, sprint planning, coding, testing, and independent review.
• 2. Build core subsystems (state layer, shader compiler/interpreter, rasterizer) by Sprint 8.
• 3. Run local tests; passes climbed from 64 to 1,334 by Sprint 13.
• 4. Rebuild captured source fresh and run custom probes to verify rendering of a triangle and simple shaders.
• 5. Identify and fix bugs through code reviews (e.g., matrix stride, compressed‑format query, negative zero division).
• 6. Detect workflow issues (e.g., automated operator decisions, test bypassing public entry point).
• 7. Document findings and lessons in a post‑mortem.
• 8. Publish article summarizing results and five problematic checks.

Доказательства

Our agents' tests went from 64 passes to 1,334 while the renderer stayed broken.

Источник

In Favur's 296-hour attempt at a Terminal-Bench WebGL renderer, local test passes rose from 64 to 1,334 while the last internal conformance results stood at 13 of 672 WebGL1 cases and 20 of 2,598 WebGL2 cases.

Источник

The renderer could draw a plain red triangle, and every stage behind it is real.

Источник