The dead probe that still billed five minutes

Description and use

A probe in the publishing pipeline that failed silently added 5 minutes of hidden latency. Setting a 60‑second timeout reduced the hidden cost and improved auditability.

Problem

An optional health‑check probe in a publishing pipeline kept waiting 300 seconds for a response that never arrived, silently adding five minutes of latency to every run.

Outcome

Reduced hidden latency from 5 minutes to 1 minute per run, improving auditability and preventing silent cost accumulation.

Workflow

• Invoke probe to query search engine
• Wait up to 300 seconds for response
• Record receipt with probe status
• If probe fails, log exit 1 and stack trace
• Continue pipeline
• After discovery, set probe timeout to 60 seconds
• Update receipt to include timeout reason

Evidence

Today the probe's invocation got a time-box: 60 seconds, down from 300, with a timeout reason written into the receipt when it fires.

Source