Building TrueSRE: Autonomous Multi‑Agent Incident Remediation with TrueForge, FastMCP, and Daytona

Description and use

TrueSRE is an autonomous multi‑agent platform that diagnoses and remediates Kubernetes outages in under 20 seconds, cutting MTTR from 45 minutes to <20 seconds, while ensuring safety with sandboxing and cryptographic HITL gates.

Problem

On‑call SREs spend excessive time manually diagnosing and remediating Kubernetes outages, incurring high MTTR and financial loss.

Outcome

TrueSRE reduced MTTR from 45 minutes to under 20 seconds for a real SEV‑1 OOMKilled outage, while ensuring safety through sandboxing and cryptographic HITL gates.

Workflow

• 1. Alert from Prometheus triggers TrueSRE
• 2. TrueForge swarm dispatches agents (Telemetry Investigator, RCA Reasoner, Remediation Planner, etc.)
• 3. Agents query Prometheus, Loki, and Git via FastMCP
• 4. Bayesian reasoner ranks hypotheses and selects 92% confident root cause
• 5. Remediation Planner drafts rollback runbook and runs Daytona sandbox dry‑run
• 6. Operator approves via HITL gate, receiving HMAC‑SHA256 token
• 7. FastMCP executes rollback, re‑queries Prometheus for verification
• 8. Post‑mortem scribe auto‑generates markdown report and stores embeddings

Evidence

Total MTTR: Under 20 seconds.

Source

TrueSRE takes outages from detection to verified resolution in under 25 seconds.

Source