My grader got every real answer right. Four red teams still broke it.

Описание и применение

Грейдер проверяет матрицу квантового схемы и сравнивает с референсом, достигая 100% точности на честных ответах, но изначально пропускал 27 атакующих схем. После четырёх раундов «red‑team» и изменений дизайна все найденные уязвимости исправлены, но полный проход «red‑team» ещё не подтверждён.

Задача

Ensuring a quantum circuit grader correctly validates AI-generated circuits while resisting adversarial manipulation.

Результат

The grader achieves perfect accuracy on honest inputs (90/90) but initially failed to resist 27 adversarial passes; after iterative red‑team testing and design changes, all identified bugs are fixed and regression tests added, though a full red‑team pass is pending.

Рабочий процесс

• Build grader that computes full matrix and compares to reference
• Run Sonnet and Haiku models on 45 tasks
• Record 45/45 pass for Sonnet, 28/45 for Haiku
• Identify 13 wrong “success” claims in Haiku
• Add post‑hoc OPENQASM 2.0 header check
• Red‑team rounds 1–4: each round identifies and fixes new bypasses
• Change design to OS sandbox, no file writes, stdin closed
• Fix bug class by inspecting gate functionality, not name
• Re‑grade all model runs after each patch
• Maintain regression tests for fixed bugs

Доказательства

45 tasks: Bell states, controlled rotations, a 3-qubit QFT, phase oracles, "use only CNOTs" constraints. 30 pilot tasks plus 15 blind ones written before any grader run on them.

Источник

45/45 for Sonnet, 28/45 for Haiku.

Источник

After every round I re‑graded both model runs, and not one verdict changed.

Источник