A Free Model Is Not a Pure Function: 200 Calls, 48 Hours, One Drifting Output

Description and use

Automating log conversion with a free LLM produces inconsistent JSON outputs that break downstream scripts. Adding a field‑validation layer and retry logic raises reliability from 61 % to 92 %.

Problem

Automating server log conversion with a free LLM produced inconsistent JSON outputs, causing downstream crashes.

Outcome

Free LLM outputs drifted 39% of the time, breaking a simple JSON parser. A validator that checks for required fields and retries improved reliability to 92%.

Workflow

• 1. Define 10 representative prompts for log tasks.
• 2. Run each prompt 20 times over 48 h via hourly cron, recording output, timestamp, latency, length.
• 3. Store metadata in CSV for time‑of‑day analysis.
• 4. Compute hash and similarity metrics per prompt.
• 5. Analyze drift: overall 61% exact matches, higher for summarization, lower for JSON and numeric prompts.
• 6. Attempt aggressive normalization (lowercase, strip punctuation) – improves to 78% but breaks JSON.
• 7. Apply strict JSON schema extraction – mitigates key‑order but fails on renamed keys.
• 8. Conclude free model is not deterministic; add deterministic validator that checks field presence and retries up to 3 times.
• 9. Re‑run test; success rate improves to 92%.
• 10. Document time‑of‑day drift and recommend defining acceptable drift metrics before testing.

Evidence

Across all 200 calls, only 61% of outputs were identical to the first output for the same prompt.

Source

The real lesson is that a free model is not a pure function.

Source

That simple retry changed my success rate from 61% to 92% in a follow-up test.

Source