I built an AI agent for production incidents. The interesting part is when it refuses to act

说明与用途

一个用于生产事件诊断的 AI 代理，能够快速识别问题，并仅在置信度和风险阈值满足时执行操作，防止不安全的更改。

问题

Automating incident response risks unintended destructive actions when an AI agent acts without sufficient confidence or oversight.

结果

Across 14 live incidents, the agent rarely altered production, correctly diagnosing issues in 8–30 s and preventing unsafe actions. It identified misconfigurations, missing IAM permissions, and unnecessary rollbacks, improving incident diagnosis speed while maintaining safety.

工作流

• Receive alert via Pub/Sub
• Collect logs from Cloud Logging and metrics from Cloud Monitoring
• Analyze logs with Gemini 3.5 to diagnose issue
• Determine risk tier based on confidence and blast radius
• If tier is OBSERVE: log findings only
• If tier is SUGGEST: recommend action to engineer
• If tier is APPROVE: stage action awaiting explicit approval
• If tier is ACT: execute action and verify via independent HTTP probe
• Log outcome and update Firestore

证据

Fourteen live incidents against a deliberately fragile Cloud Run service.

来源

In one incident it reached the ACT tier and ran a real command against production. The command failed — a missing IAM permission I had not anticipated.

来源

The agent did it in 8 to 30 seconds, every time, with the file, the line number, the occurrence count, and the offending revision.

来源