48 Hours of Field Notes on a Free AI Stack: What Broke, What Held, and What I'd Repeat

说明与用途

在免费 AI 堆栈上进行的 48 小时现场实验揭示了冷启动、速率限制和细微输出漂移问题。双时钟、形状签名和中途人工审查帮助快速检测和修复错误。

问题

Understanding real‑world behavior of a free AI stack outside of load tests, especially under idle periods, rate limits, and subtle output drift.

结果

Field logging with dual clocks and shape signatures turned 192 messy runs into a readable story, revealing cold start delays, rate‑limit bursts, and subtle output drift that would have gone unnoticed by load tests alone.

工作流

• Set up free server and model access; schedule digest job every 15 min.
• Append each run to JSONL log with run_id, stage, outcome, wall time, monotonic time, latency, shape signature, and note.
• Use shape_signature to detect output drift (keys, length bucket, starts_with_list).
• Log latency and compare wall vs monotonic to spot cold starts.
• Plot failures against wall clock to detect rate‑limit bursts; shift schedule with jitter.
• Perform human review at 24 h to catch drift missed by gate.
• Adjust retry logic, timeouts, and backoff based on findings.
• Re‑prompt with pinned examples when drift detected.
• Reconcile log order using run_id after restarts.
• Document triage table for symptom → evidence → verdict → action.
• Repeat dual clocks, shape signature, fixed cadence, and mid‑experiment review in future experiments.
• Avoid hash gate and over‑reliance on load tests alone.

证据

After 48 hours I had 192 runs, 14 failures, and three lessons I would happily repeat.

来源

The first call after twenty idle minutes took 34 seconds, while my client timeout was set to 10 seconds, so the job failed and the retry fired immediately.

来源

At a fixed minute past the hour, the provider started returning 429s in a burst and then went silent, and my exponential backoff made it worse.

来源

The model returned valid JSON with correct keys, and the summary was subtly wrong in structure because it had switched from prose to a terse list.

来源