My Raspberry Pi's Clock Drifted 9 Minutes. My AI Agent Silently Accepted 401s for Six Days.

说明与用途

在 Raspberry Pi 上，时钟漂移了 9 分钟，导致 AI 代理在六天内收到 401 错误。检测到问题后，修复了时间同步，添加了漂移检查，将 401 视为致命错误，并将监控从仅检查进程存活改为验证结果。

问题

An AI agent on a Raspberry Pi kept sending signed requests to an API, but the Pi’s clock drifted 9 minutes, causing all requests to be rejected with 401 Unauthorized for six days.

结果

Implemented clock drift detection, corrected time sync, treated auth errors as fatal, and shifted monitoring from liveness to outcome verification, preventing silent failures and saving API quota.

工作流

• 1. Detect clock drift by comparing Pi time to an HTTP Date header and alert if >60s.
• 2. Restart systemd-timesyncd or enable NTP to resync clock.
• 3. Modify agent error handling: treat 401 as fatal and stop retries.
• 4. Update watchdog to verify artifacts (email message ID, output files) instead of only process liveness.
• 5. Add periodic clock drift checks (every 15 minutes).

证据

The Pi fell back to the last time it remembered — the fake "epoch-ish" time it uses before first sync, adjusted by a saved timestamp file. It came up roughly 9 minutes fast.

来源

I added a tiny health check to my watchdog: ... if drift > max_drift_seconds: alert(...).

来源