My Staging Agent and My Production Agent Were Different Agents

说明与用途

在生产和预发布环境之间同步配置导致同一 AI 代理因环境变量漂移而产生不同的响应。

问题

The same AI agent code produced different customer-facing responses in staging and production due to silent configuration drift.

结果

Configuration drift went unnoticed for 63 days, leading to incorrect refund windows and pricing in production. After implementing the five-step mitigation, drift is detected immediately and customer-facing behavior remains consistent.

工作流

• 1. Log a hash of resolved config at boot and job start.
• 2. Make business keys required; fail startup if missing.
• 3. Render config from a single versioned file during deploy; fail on unknown keys.
• 4. Add a runbook command to diff config between environments before deploy.
• 5. Deploy a canary that runs fixed inputs and compares outputs to golden results.

证据

None of those values were in the repo. They were in /etc/agent/prod.env

来源

The agent never threw an exception. That's the part that still bothers me.

来源

I did the obvious thing: wrote a sync script to copy staging's env onto prod

来源