A run history cannot show you a run it never created

Description and use

Run histories miss silent failures because they only record transport success. A reconciliation framework that logs events, receipts, and send outcomes, classifies outcomes, and reports a bounded failure rate using the Wilson interval, provides accurate reliability metrics.

Problem

Event‑driven systems can report a successful run even when the work never actually happened, because logs only record transport success and cannot capture silent failures or missing runs.

Outcome

Implementing this reconciliation framework exposes silent failures that run histories hide, enabling accurate reliability metrics and preventing misleading claims of zero failure rates.

Workflow

• 1. Emit an event with a unique run_id and metadata (wf, platform, seq, fired_at).
• 2. Each workflow step posts a receipt to a controlled destination, echoing the event and adding step info; the destination stamps received_at.
• 3. The controller records the send outcome (accepted, refused, error type).
• 4. Reconcile by matching receipts to the original event’s run_id and comparing against expected outputs.
• 5. Classify each event into one of seven outcomes based on receipt count and send status.
• 6. Compute silent failure rate as (missed + partial) / (runs accepted and expected to produce output).
• 7. Report the rate with denominator, Wilson interval, as‑of date, workload, and whether the figure is live.
• 8. Publish all seven outcome counts for transparency.
• 9. Use L1, L2, or L3 conformance levels to determine measurement rigor.

Evidence

A run history cannot show you a run it never created

Source

Events refused at send are excluded from the numerator and from the denominator, and reported separately.

Source

Use the Wilson score interval, not the normal approximation.

Source