My Nightly Job Ran at 3 PM. Then the Disk Forgot, and the Model Returned Prose.

Description and use

A cron‑scheduled summarization job ran at the wrong time, lost its checkpoint due to an ephemerally mounted disk, and received prose instead of JSON from a free LLM. Solutions include a clock drift check, remote checkpoint storage, and robust JSON validation.

Problem

A scheduled summarization job on a free server ran at the wrong time, lost its checkpoint data, and received malformed output from a free model endpoint.

Outcome

The job now runs at the intended time, survives server restarts by persisting checkpoints remotely, and reliably parses JSON from the model, avoiding failures caused by malformed output.

Workflow

• 1. Add a clock drift check that compares local time to an HTTP Date header and exits if drift > 60s.
• 2. Integrate the clock check into the main script before any work is done.
• 3. Replace local checkpoint file with a remote checkpoint by sending a PUT request after each item.
• 4. Store the checkpoint URL in a secure location (e.g., object storage or a tiny endpoint).
• 5. Wrap model responses in a try/except block to catch JSON parsing errors.
• 6. If parsing fails, log the raw response and skip the item or retry.
• 7. Continue processing remaining items and update the remote checkpoint after each successful summary.

Evidence

The free server had been suspended overnight, and when it resumed, its clock was still living in the afternoon.

Source

The disk was erased on restart; the writable disk is ephemerally like /tmp.

Source

The model returned prose instead of JSON, breaking the JSON parsing line.

Source