My Raspberry Pi's Clock Drifted 9 Minutes. My AI Agent Silently Accepted 401s for Six Days.

Описание и применение

На Raspberry Pi часы отстали на 9 минут, из‑за чего агент, подписывающий запросы, получал 401‑ошибки в течение шести дней. После обнаружения проблемы исправили синхронизацию времени, добавили проверку дрейфа, сделали 401‑ошибки фатальными и изменили мониторинг на проверку результатов, а не только живучести процессов.

Задача

An AI agent on a Raspberry Pi kept sending signed requests to an API, but the Pi’s clock drifted 9 minutes, causing all requests to be rejected with 401 Unauthorized for six days.

Результат

Implemented clock drift detection, corrected time sync, treated auth errors as fatal, and shifted monitoring from liveness to outcome verification, preventing silent failures and saving API quota.

Рабочий процесс

• 1. Detect clock drift by comparing Pi time to an HTTP Date header and alert if >60s.
• 2. Restart systemd-timesyncd or enable NTP to resync clock.
• 3. Modify agent error handling: treat 401 as fatal and stop retries.
• 4. Update watchdog to verify artifacts (email message ID, output files) instead of only process liveness.
• 5. Add periodic clock drift checks (every 15 minutes).

Доказательства

The Pi fell back to the last time it remembered — the fake "epoch-ish" time it uses before first sync, adjusted by a saved timestamp file. It came up roughly 9 minutes fast.

Источник

I added a tiny health check to my watchdog: ... if drift > max_drift_seconds: alert(...).

Источник