A 90‑Minute Spike for Health Checks That Can Fail

Description and use

A 90‑minute spike ensures /ready returns 503 when the store is down and 200 after recovery, preventing traffic to unhealthy services.

Problem

Health‑check endpoints often return 200 even when critical dependencies are down, causing load balancers to route traffic to unhealthy services.

Outcome

After a 90‑minute spike, the service correctly reports 503 on /ready when the store is down and recovers to 200 once the store is available, preventing traffic to unhealthy instances.

Workflow

• Set up a store stub that raises DependencyDown when STORE_DOWN=1
• Create a baseline handler that always returns 200 for /health and /ready
• Write kill tests that start the app with STORE_DOWN=1 and verify /ready stays 200
• Run tests for 90 minutes, cycling through constrained agent passes and repair passes
• Stop the spike after 90 minutes and document the ship‑or‑kill decision

Evidence

Unconstrained coding agents skip checks in health routes. They return 200 because the process started.

Source

Kill if /ready stays 2xx while a required store is down.

Source