Compare Against the Schema They Shipped, Not the One You Expected

Description and use

The exec call harness was incorrectly comparing against a frozen JSON that conflicted with the provider’s schema, causing false mismatches. The fix updated the expected object to include only required fields and kept the comparison strict.

Problem

A harness expected a specific JSON object for an exec tool call, but the provider’s schema allowed additional optional fields, causing a false mismatch error.

Outcome

The harness now correctly identifies mismatches due to schema conflicts rather than false positives, preserving control integrity.

Workflow

• 1. Read provider’s exec schema to understand required fields.
• 2. Define a constant intent value for the harness.
• 3. Build the expected object with command and the constant intent.
• 4. Validate that the expected object has exactly two keys and matches the provider’s required set.
• 5. Compare the model’s actual JSON to the expected object using canonical JSON serialization.
• 6. If mismatch, report EXEC_ARGUMENTS_MISMATCH without altering the comparison logic.

Evidence

The provider's exec schema — from the compiled artifact I ran, @truefoundry/[email protected], dist/core/sandbox/Sandbox.mjs

Source

The expected object gate also got stricter…

Source