48 Hours of Field Notes on a Free AI Stack: What Broke, What Held, and What I'd Repeat

Description and use

A 48‑hour field experiment on a free AI stack uncovered cold starts, rate‑limit bursts, and subtle output drift. Dual clocks, shape signatures, and a mid‑run human review enabled rapid detection and remediation.

Problem

Understanding real‑world behavior of a free AI stack outside of load tests, especially under idle periods, rate limits, and subtle output drift.

Outcome

Field logging with dual clocks and shape signatures turned 192 messy runs into a readable story, revealing cold start delays, rate‑limit bursts, and subtle output drift that would have gone unnoticed by load tests alone.

Workflow

• Set up free server and model access; schedule digest job every 15 min.
• Append each run to JSONL log with run_id, stage, outcome, wall time, monotonic time, latency, shape signature, and note.
• Use shape_signature to detect output drift (keys, length bucket, starts_with_list).
• Log latency and compare wall vs monotonic to spot cold starts.
• Plot failures against wall clock to detect rate‑limit bursts; shift schedule with jitter.
• Perform human review at 24 h to catch drift missed by gate.
• Adjust retry logic, timeouts, and backoff based on findings.
• Re‑prompt with pinned examples when drift detected.
• Reconcile log order using run_id after restarts.
• Document triage table for symptom → evidence → verdict → action.
• Repeat dual clocks, shape signature, fixed cadence, and mid‑experiment review in future experiments.
• Avoid hash gate and over‑reliance on load tests alone.

Evidence

After 48 hours I had 192 runs, 14 failures, and three lessons I would happily repeat.

Source

The first call after twenty idle minutes took 34 seconds, while my client timeout was set to 10 seconds, so the job failed and the retry fired immediately.

Source

At a fixed minute past the hour, the provider started returning 429s in a burst and then went silent, and my exponential backoff made it worse.

Source

The model returned valid JSON with correct keys, and the summary was subtly wrong in structure because it had switched from prose to a terse list.

Source