My Staging Agent and My Production Agent Were Different Agents

Описание и применение

Синхронизация конфигураций между средами привела к тому, что один и тот же AI‑агент выдавал разные ответы в продакшене и стенде из‑за незаметного сдвига переменных окружения.

Задача

The same AI agent code produced different customer-facing responses in staging and production due to silent configuration drift.

Результат

Configuration drift went unnoticed for 63 days, leading to incorrect refund windows and pricing in production. After implementing the five-step mitigation, drift is detected immediately and customer-facing behavior remains consistent.

Рабочий процесс

• 1. Log a hash of resolved config at boot and job start.
• 2. Make business keys required; fail startup if missing.
• 3. Render config from a single versioned file during deploy; fail on unknown keys.
• 4. Add a runbook command to diff config between environments before deploy.
• 5. Deploy a canary that runs fixed inputs and compares outputs to golden results.

Доказательства

None of those values were in the repo. They were in /etc/agent/prod.env

Источник

The agent never threw an exception. That's the part that still bothers me.

Источник

I did the obvious thing: wrote a sync script to copy staging's env onto prod

Источник