Building TrueSRE: Autonomous Multi‑Agent Incident Remediation with TrueForge, FastMCP, and Daytona

说明与用途

TrueSRE 是一款自主多代理平台，能够在不到 20 秒内诊断并修复 Kubernetes 故障，将 MTTR 从 45 分钟降至不到 20 秒，同时通过沙箱和加密 HITL 门实现安全保障。

问题

On‑call SREs spend excessive time manually diagnosing and remediating Kubernetes outages, incurring high MTTR and financial loss.

结果

TrueSRE reduced MTTR from 45 minutes to under 20 seconds for a real SEV‑1 OOMKilled outage, while ensuring safety through sandboxing and cryptographic HITL gates.

工作流

• 1. Alert from Prometheus triggers TrueSRE
• 2. TrueForge swarm dispatches agents (Telemetry Investigator, RCA Reasoner, Remediation Planner, etc.)
• 3. Agents query Prometheus, Loki, and Git via FastMCP
• 4. Bayesian reasoner ranks hypotheses and selects 92% confident root cause
• 5. Remediation Planner drafts rollback runbook and runs Daytona sandbox dry‑run
• 6. Operator approves via HITL gate, receiving HMAC‑SHA256 token
• 7. FastMCP executes rollback, re‑queries Prometheus for verification
• 8. Post‑mortem scribe auto‑generates markdown report and stores embeddings

证据

Total MTTR: Under 20 seconds.

来源

TrueSRE takes outages from detection to verified resolution in under 25 seconds.

来源