Your Self-Hosted Agent Will Break at 3 a.m. Here Is What Should Happen Next

Description and use

Implemented bounded retry loops and a structured remediation vocabulary, reducing silent nighttime failures to a single log line and achieving a 93 % automated repair rate.

Problem

Self-hosted AI agents often fail silently at night, causing production outages that go unnoticed until a customer or cron job reports them.

Outcome

Implemented bounded retry loops and a structured remediation vocabulary, reducing silent nighttime failures to a single log line and achieving a 93 % automated repair rate.

Workflow

• 1. Deploy a lightweight watchdog script that checks each critical container’s state every few minutes and starts an emergency container only if no healthy backend is available, with a single retry limit and an exclusive lock to avoid duplicate bridges.
• 2. Run a post-deploy repair loop immediately after each deployment that performs HTTP and visual checks, applies a small set of known remediations (restart container, restart proxy, clear cache, run migration step), and retries up to three times before escalating.
• 3. Execute a broad night repair script that scans for known failure patterns (e.g., stuck queue, exhausted connection pool, non-zero exit code, stale search index) and applies the appropriate fix for each, logging every action and maintaining an immutable incident record.
• 4. Configure escalation to a human by sending a push notification with context (app, failed check, attempts, last fix) when the retry budget is exhausted.
• 5. Continuously add new remediation patterns as incidents occur, keeping the script size proportional to real incidents rather than design speculation.

Evidence

Across the last months my operations cockpit counts 1,354 tasks completed without a human touching them, out of 1,873 attempted, a 93 percent rate.

Source

The first rule is a hard limit on retries. Every repair attempt gets a small budget. When the budget is spent, the script stops, marks the incident open, and wakes a human.

Source

The phrase "known remediation" is doing real work. It does not mean "let the agent improvise".

Source