From Telemetry to Automated Incident Response: My DevOps & Observability Journey

Description and use

Collecting metrics, logs, and traces via OpenTelemetry, visualizing in Grafana, configuring 5xx alerts, automatically creating incidents, and passing evidence to an AI coding agent Codex CLI enabled rapid detection, investigation, and remediation of errors in a FastAPI application.

Problem

A basic FastAPI order‑tracking API can fail in production without observability, making troubleshooting manual, error‑prone, and slow.

Outcome

The observability stack enabled automated detection of 5xx errors, automated incident creation, evidence collection, and AI‑driven code fixes, reducing manual debugging time and improving reliability.

Workflow

• Instrument FastAPI with OpenTelemetry to emit HTTP route, status, logs, traces, and metrics
• Deploy OpenTelemetry Collector to receive OTLP data and route to Prometheus, Loki, and Tempo
• Configure Grafana to visualize signals and set alert rules for 5xx responses
• Implement an incident‑response service exposing POST /alerts to receive Grafana alerts and gather evidence
• Integrate Codex CLI as a coding agent that receives incident context, investigates, and applies fixes
• Restart and verify the application after remediation

Evidence

The project started as a basic Order Tracker application backed by SQLite.

Source

The final architecture introduced several components: OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana.

Source

After telemetry was available, I configured Grafana alerting for server‑side failures. The goal was to detect 5xx responses.

Source

The service exposes: POST /alerts. When Grafana sends an alert, the responder stores the incident information and gathers useful evidence.

Source

The most interesting part of the exercise was connecting the incident workflow to a coding agent.

Source