48-Hour Field Notes: The Free Model Returned Valid JSON, and My Schema Said Nothing

Description and use

During a 48‑hour field test, a free LLM returned empty arrays that the schema accepted as valid. Adding minItems:1 and a pre‑flight probe caught configuration errors and improved reliability.

Problem

A nightly extraction job using a free model tier returned empty arrays because the schema did not enforce a minimum item count, leading to silent failures and wasted retries.

Outcome

Adding minItems:1 prevented silent empty results, the pre‑flight probe caught configuration and environment issues early, and the refined retry taxonomy reduced wasted compute and avoided silent failures.

Workflow

• 1. Define JSON schema with required fields and minItems:1 for records.
• 2. Run a pre‑flight probe: send ten identical requests, ambiguous blob, oversized blob, wrong model name, idle window test, and kill worker test.
• 3. If probe passes, launch nightly job.
• 4. On validation failure, perform one repair attempt then fail; do not retry content failures.
• 5. On transport or rate_limited failures, retry with bounded backoff.
• 6. On config failures (401/403/404), stop the run immediately.
• 7. Log all events and preserve journal outside the worker’s disk.

Evidence

The model, faced with a blob it could not parse confidently, made a reasonable choice and returned nothing rather than inventing line items.

Source

The one-keyword fix is "minItems": 1, and it is the single highest-value line in the whole article.

Source

A mistyped model name produced a 4xx on every request. My retry wrapper treated everything that was not a 200 as transient, so the job cheerfully backed off and retried for hours.

Source