My grader got every real answer right. Four red teams still broke it.

说明与用途

该评估器通过比较完整的量子电路矩阵与参考值来验证 AI 生成的电路，诚实输入时准确率为 100%，但最初允许 27 个攻击性通过。经过四轮红队测试和设计更改后，已修复所有已知漏洞，但尚未完成完整红队通过验证。

问题

Ensuring a quantum circuit grader correctly validates AI-generated circuits while resisting adversarial manipulation.

结果

The grader achieves perfect accuracy on honest inputs (90/90) but initially failed to resist 27 adversarial passes; after iterative red‑team testing and design changes, all identified bugs are fixed and regression tests added, though a full red‑team pass is pending.

工作流

• Build grader that computes full matrix and compares to reference
• Run Sonnet and Haiku models on 45 tasks
• Record 45/45 pass for Sonnet, 28/45 for Haiku
• Identify 13 wrong “success” claims in Haiku
• Add post‑hoc OPENQASM 2.0 header check
• Red‑team rounds 1–4: each round identifies and fixes new bypasses
• Change design to OS sandbox, no file writes, stdin closed
• Fix bug class by inspecting gate functionality, not name
• Re‑grade all model runs after each patch
• Maintain regression tests for fixed bugs

证据

45 tasks: Bell states, controlled rotations, a 3-qubit QFT, phase oracles, "use only CNOTs" constraints. 30 pilot tasks plus 15 blind ones written before any grader run on them.

来源

45/45 for Sonnet, 28/45 for Haiku.

来源

After every round I re‑graded both model runs, and not one verdict changed.

来源