Compare Against the Schema They Shipped, Not the One You Expected

Описание и применение

Валидация вызовов инструментов в системе интеграции была настроена слишком строго, что приводило к ложным ошибкам. После исправления хранилище теперь корректно сравнивает аргументы с обязательными полями схемы поставщика, избегая неверных срабатываний.

Задача

A harness expected a specific JSON structure for an exec tool call, but the provider’s schema allowed optional fields, causing a mismatch and false failure.

Результат

The harness now correctly validates exec calls against the provider’s schema, avoiding false failures and ensuring that only intentional deviations are flagged.

Рабочий процесс

• 1. Inspect the provider’s exec tool schema to understand required and optional fields.
• 2. Define the harness’s expected object, ensuring it includes only the required fields and any harness‑specific constraints.
• 3. Validate the expected object against the provider’s schema before use.
• 4. Compare the model’s actual arguments to the expected object using canonical JSON serialization.
• 5. If a mismatch occurs, determine which side (provider schema or harness expectation) is incorrect before adjusting the harness.
• 6. Update the harness to enforce the correct key set and constant intent value.
• 7. Retest the harness to confirm no false mismatches occur.

Доказательства

The provider’s exec schema defined required and optional fields, but the harness froze a stricter contract.

Источник

The harness was updated to set a constant intent value and require exactly two keys.

Источник