I built an AI agent for production incidents. The interesting part is when it refuses to act

Описание и применение

Создан агент AI для диагностики инцидентов в Google Cloud Run, который быстро выявляет проблемы и только выполняет действия при достаточной уверенности, предотвращая нежелательные изменения.

Задача

Automating incident response risks unintended destructive actions when an AI agent acts without sufficient confidence or oversight.

Результат

Across 14 live incidents, the agent rarely altered production, correctly diagnosing issues in 8–30 s and preventing unsafe actions. It identified misconfigurations, missing IAM permissions, and unnecessary rollbacks, improving incident diagnosis speed while maintaining safety.

Рабочий процесс

• Receive alert via Pub/Sub
• Collect logs from Cloud Logging and metrics from Cloud Monitoring
• Analyze logs with Gemini 3.5 to diagnose issue
• Determine risk tier based on confidence and blast radius
• If tier is OBSERVE: log findings only
• If tier is SUGGEST: recommend action to engineer
• If tier is APPROVE: stage action awaiting explicit approval
• If tier is ACT: execute action and verify via independent HTTP probe
• Log outcome and update Firestore

Доказательства

Fourteen live incidents against a deliberately fragile Cloud Run service.

Источник

In one incident it reached the ACT tier and ran a real command against production. The command failed — a missing IAM permission I had not anticipated.

Источник

The agent did it in 8 to 30 seconds, every time, with the file, the line number, the occurrence count, and the offending revision.

Источник