My Raspberry Pi's Clock Drifted 9 Minutes. My AI Agent Silently Accepted 401s for Six Days.

Description and use

On a Raspberry Pi, the clock drifted 9 minutes, causing an AI agent to receive 401 errors for six days. After detecting the issue, time sync was fixed, a drift check was added, 401s were treated as fatal, and monitoring shifted to outcome verification instead of just process liveness.

Problem

An AI agent on a Raspberry Pi kept sending signed requests to an API, but the Pi’s clock drifted 9 minutes, causing all requests to be rejected with 401 Unauthorized for six days.

Outcome

Implemented clock drift detection, corrected time sync, treated auth errors as fatal, and shifted monitoring from liveness to outcome verification, preventing silent failures and saving API quota.

Workflow

• 1. Detect clock drift by comparing Pi time to an HTTP Date header and alert if >60s.
• 2. Restart systemd-timesyncd or enable NTP to resync clock.
• 3. Modify agent error handling: treat 401 as fatal and stop retries.
• 4. Update watchdog to verify artifacts (email message ID, output files) instead of only process liveness.
• 5. Add periodic clock drift checks (every 15 minutes).

Evidence

The Pi fell back to the last time it remembered — the fake "epoch-ish" time it uses before first sync, adjusted by a saved timestamp file. It came up roughly 9 minutes fast.

Source

I added a tiny health check to my watchdog: ... if drift > max_drift_seconds: alert(...).

Source