The fallback chain that silently stops working

Description and use

<h1> The fallback chain that silently stops working </h1> <p>I found out my fallback chain was broken because of a bill. Not an alert, not a red row in a dashboard — the spend for that week came out lower than usual in a way that felt wrong, and when I went looking, every request had been answered by the last model in the chain. The app was fine. Latency was tolerable. Nobody complained. I had been running on the emergency route for a while and calling it production.</p> <p>The first route in the chain had been retired. What stayed with me afterwards was how loudly that should have failed. A r

Problem

<h1> The fallback chain that silently stops working </h1> <p>I found out my fallback chain was broken because of a bill. Not an alert, not a red row in a dashboard — the spend for that week came out lower than usual in a way that felt wrong, and when I went looking, every request had been answered by the last model in the chain. The app was fine. Latency was tolerable. Nobody complained. I had been running on the emergency route for a while and calling it production.</p> <p>The first route in the chain had been retired. What stayed with me afterwards was how loudly that should have failed. A r

Outcome

<h1> The fallback chain that silently stops working </h1> <p>I found out my fallback chain was broken because of a bill. Not an alert, not a red row in a dashboard — the spend for that week came out lower than usual in a way that felt wrong, and when I went looking, every request had been answered by the last model in the chain. The app was fine. Latency was tolerable. Nobody complained. I had been running on the emergency route for a while and calling it production.</p> <p>The first route in the chain had been retired. What stayed with me afterwards was how loudly that should have failed. A r

Evidence

The fallback chain that silently stops working

Source