AI On-Call Copilot with Human‑Signed Approval Gate

Description and use

An AI on‑call copilot rapidly diagnoses incidents but requires cryptographically signed human approval before any remediation, ensuring safety and auditability.

Problem

AIOps tools either overwhelm operators with alerts or act autonomously, risking hallucinated actions that can damage production.

Outcome

The system achieved fast incident diagnosis while preventing unauthorized automated remediation, ensuring every action required a cryptographically verified human approval and was fully auditable.

Workflow

• 1. Alert received from monitoring stack
• 2. Service pulls metrics, logs, and deploy history for context
• 3. LLM generates structured diagnosis and proposed action
• 4. If action is code‑level revert, opens a PR for merge approval
• 5. If action is operational (restart, rollback, scale), posts interactive approval request to Slack/Teams/email
• 6. Human approves via signed token; token is verified, single‑use, and bound to exact action
• 7. Approved action is executed via safe execFile call with allowlist validation
• 8. All steps logged to an append‑only audit log

Evidence

Every alert, diagnosis, approval decision, token consumption, and execution result is written to an append-only audit log.

Source

The LLM-proposed target is never passed to kubectl until it's matched against an explicit allowlist.

Source

Every approval request mints a cryptographically signed, single-use, short‑TTL HMAC token.

Source