It Ran Every Morning and Still Broke: Failure Modes of a Free-Tier AI Job

Description and use

A daily cron job that fetches release notes, summarizes them with an AI model, and writes the summary often ran without errors but produced incorrect or missing output. The article details five common failure modes and presents a healthcheck script that validates output size, schema, ledger entry, state consistency, and truncation warnings.

Problem

A daily cron job that fetches release notes, summarizes them with an AI model, and writes the summary failed silently on a free tier, producing incorrect or empty outputs without raising errors.

Outcome

The healthcheck script catches five common failure modes of free-tier AI jobs, turning silent failures into actionable alerts and ensuring reliable daily summaries.

Workflow

• 1. Verify latest_summary.md is non-empty and >50 chars.
• 2. Check that summary contains expected sections: Breaking, Migration, Affects.
• 3. Confirm ledger.jsonl has a recent entry with today's date.
• 4. Ensure state file (last_seen.json) matches the ledger's tag.
• 5. Scan watch.log for any truncation warnings.

Evidence

The cron log said exit 0 every morning. The summary file updated on schedule. The job was running. The output was wrong.

Source

The fix was to validate the response body against an expected shape, not just the status code.

Source

The healthcheck script verifies output size, schema, ledger entry, state consistency, and truncation warnings.

Source