Compare Against the Schema They Shipped, Not the One You Expected

说明与用途

工具调用验证的 harness 过于严格，导致误报。修正后，它现在正确地将参数与提供商的必需字段进行比较，避免了错误的匹配。

问题

A harness expected a specific JSON structure for an exec tool call, but the provider’s schema allowed optional fields, causing a mismatch and false failure.

结果

The harness now correctly validates exec calls against the provider’s schema, avoiding false failures and ensuring that only intentional deviations are flagged.

工作流

• 1. Inspect the provider’s exec tool schema to understand required and optional fields.
• 2. Define the harness’s expected object, ensuring it includes only the required fields and any harness‑specific constraints.
• 3. Validate the expected object against the provider’s schema before use.
• 4. Compare the model’s actual arguments to the expected object using canonical JSON serialization.
• 5. If a mismatch occurs, determine which side (provider schema or harness expectation) is incorrect before adjusting the harness.
• 6. Update the harness to enforce the correct key set and constant intent value.
• 7. Retest the harness to confirm no false mismatches occur.

证据

The provider’s exec schema defined required and optional fields, but the harness froze a stricter contract.

来源

The harness was updated to set a constant intent value and require exactly two keys.

来源