My Staging Agent and My Production Agent Were Different Agents

Description and use

Synchronizing configuration between environments caused the same AI agent to give different answers in production and staging due to silent environment variable drift.

Problem

The same AI agent code produced different customer-facing responses in staging and production due to silent configuration drift.

Outcome

Configuration drift went unnoticed for 63 days, leading to incorrect refund windows and pricing in production. After implementing the five-step mitigation, drift is detected immediately and customer-facing behavior remains consistent.

Workflow

• 1. Log a hash of resolved config at boot and job start.
• 2. Make business keys required; fail startup if missing.
• 3. Render config from a single versioned file during deploy; fail on unknown keys.
• 4. Add a runbook command to diff config between environments before deploy.
• 5. Deploy a canary that runs fixed inputs and compares outputs to golden results.

Evidence

None of those values were in the repo. They were in /etc/agent/prod.env

Source

The agent never threw an exception. That's the part that still bothers me.

Source

I did the obvious thing: wrote a sync script to copy staging's env onto prod

Source