I Added a Circuit Breaker to My AI Agent Setup — It Caught Three Outages This Month

Description and use

Implementing a circuit breaker in an AI agent system quickly detects unavailable nodes, stops retrying, and switches to fallback models, reducing wait times and queue errors.

Problem

When a machine in an AI agent setup goes down, agents keep sending requests to it, causing long timeouts, queue buildup, and cascading failures.

Outcome

The circuit breaker reduced queue time, prevented cascading failures, and caught three real outages in a month, improving overall system resilience.

Workflow

• 1. Define a CircuitBreaker class with closed, open, and half‑open states. 2. Set thresholds (e.g., 3 consecutive failures) and cooldown (e.g., 60 s). 3. In the request wrapper, call breaker.allow_request() before making the HTTP call. 4. On success, call breaker.record_success(); on failure, call breaker.record_failure(). 5. When the breaker is open, raise CircuitOpen to fail fast. 6. Configure the router to consult the breaker before routing a task. 7. Use aggressive timeouts (5 s) to distinguish dead machines from slow responses. 8. Pair the breaker with existing fallbacks (e.g., use CPU model if GPU is down).

Evidence

Since I added it, it's caught three real outages this month.

Source

One of my machines goes down — a reboot, a driver update, someone trips over a power cable — and the agents that depend on it just keep firing requests at it.

Source