Compare Against the Schema They Shipped, Not the One You Expected

Description and use

The harness for validating tool calls was too strict, causing false failures. After correction, it now correctly compares arguments against the provider’s required schema fields, preventing incorrect mismatches.

Problem

A harness expected a specific JSON structure for an exec tool call, but the provider’s schema allowed optional fields, causing a mismatch and false failure.

Outcome

The harness now correctly validates exec calls against the provider’s schema, avoiding false failures and ensuring that only intentional deviations are flagged.

Workflow

• 1. Inspect the provider’s exec tool schema to understand required and optional fields.
• 2. Define the harness’s expected object, ensuring it includes only the required fields and any harness‑specific constraints.
• 3. Validate the expected object against the provider’s schema before use.
• 4. Compare the model’s actual arguments to the expected object using canonical JSON serialization.
• 5. If a mismatch occurs, determine which side (provider schema or harness expectation) is incorrect before adjusting the harness.
• 6. Update the harness to enforce the correct key set and constant intent value.
• 7. Retest the harness to confirm no false mismatches occur.

Evidence

The provider’s exec schema defined required and optional fields, but the harness froze a stricter contract.

Source

The harness was updated to set a constant intent value and require exactly two keys.

Source