Gate AI Workflow Acceptance on Mandatory Checks

Description and use

<p>An agent can configure most of a workflow correctly and still leave a required approval broken. If your evaluation reports only an average score, that failure can disappear inside an encouraging number.</p> <p>OpenAI's October 6 publication, <a href="https://openai.com/index/advancing-computer-use-with-ironclad/" rel="noopener noreferrer">Advancing computer use with Ironclad</a>, makes this a concrete engineering problem. The researchers worked with practitioners to define 11 tasks across legal, commercial and procurement work, with 8 to 50 evaluation criteria per task.</p> <p>OpenAI report

Problem

<p>An agent can configure most of a workflow correctly and still leave a required approval broken. If your evaluation reports only an average score, that failure can disappear inside an encouraging number.</p> <p>OpenAI's October 6 publication, <a href="https://openai.com/index/advancing-computer-use-with-ironclad/" rel="noopener noreferrer">Advancing computer use with Ironclad</a>, makes this a concrete engineering problem. The researchers worked with practitioners to define 11 tasks across legal, commercial and procurement work, with 8 to 50 evaluation criteria per task.</p> <p>OpenAI report

Outcome

<p>An agent can configure most of a workflow correctly and still leave a required approval broken. If your evaluation reports only an average score, that failure can disappear inside an encouraging number.</p> <p>OpenAI's October 6 publication, <a href="https://openai.com/index/advancing-computer-use-with-ironclad/" rel="noopener noreferrer">Advancing computer use with Ironclad</a>, makes this a concrete engineering problem. The researchers worked with practitioners to define 11 tasks across legal, commercial and procurement work, with 8 to 50 evaluation criteria per task.</p> <p>OpenAI report

Evidence

Gate AI Workflow Acceptance on Mandatory Checks

Source