Building TrueSRE: Autonomous Multi‑Agent Incident Remediation with TrueForge, FastMCP, and Daytona

Описание и применение

TrueSRE – автономная платформа с многопользовательскими агентами, которая в течение 20 секунд диагностирует и устраняет сбои в Kubernetes, снижая MTTR с 45 минут до <20 секунд, при этом обеспечивая безопасность через изолированные среды и криптографические токены.

Задача

On‑call SREs spend excessive time manually diagnosing and remediating Kubernetes outages, incurring high MTTR and financial loss.

Результат

TrueSRE reduced MTTR from 45 minutes to under 20 seconds for a real SEV‑1 OOMKilled outage, while ensuring safety through sandboxing and cryptographic HITL gates.

Рабочий процесс

• 1. Alert from Prometheus triggers TrueSRE
• 2. TrueForge swarm dispatches agents (Telemetry Investigator, RCA Reasoner, Remediation Planner, etc.)
• 3. Agents query Prometheus, Loki, and Git via FastMCP
• 4. Bayesian reasoner ranks hypotheses and selects 92% confident root cause
• 5. Remediation Planner drafts rollback runbook and runs Daytona sandbox dry‑run
• 6. Operator approves via HITL gate, receiving HMAC‑SHA256 token
• 7. FastMCP executes rollback, re‑queries Prometheus for verification
• 8. Post‑mortem scribe auto‑generates markdown report and stores embeddings

Доказательства

Total MTTR: Under 20 seconds.

Источник

TrueSRE takes outages from detection to verified resolution in under 25 seconds.

Источник