A run history cannot show you a run it never created

Описание и применение

Система логов не фиксирует события, которые не произошли, поэтому отчёты о выполнениях скрывают беззвучные ошибки. Предлагается схема, где каждое событие записывается, подтверждается шагами и классифицируется, а затем рассчитывается надёжность с помощью интервала Вильсона.

Задача

Event‑driven systems can report a successful run even when the work never actually happened, because logs only record transport success and cannot capture silent failures or missing runs.

Результат

Implementing this reconciliation framework exposes silent failures that run histories hide, enabling accurate reliability metrics and preventing misleading claims of zero failure rates.

Рабочий процесс

• 1. Emit an event with a unique run_id and metadata (wf, platform, seq, fired_at).
• 2. Each workflow step posts a receipt to a controlled destination, echoing the event and adding step info; the destination stamps received_at.
• 3. The controller records the send outcome (accepted, refused, error type).
• 4. Reconcile by matching receipts to the original event’s run_id and comparing against expected outputs.
• 5. Classify each event into one of seven outcomes based on receipt count and send status.
• 6. Compute silent failure rate as (missed + partial) / (runs accepted and expected to produce output).
• 7. Report the rate with denominator, Wilson interval, as‑of date, workload, and whether the figure is live.
• 8. Publish all seven outcome counts for transparency.
• 9. Use L1, L2, or L3 conformance levels to determine measurement rigor.

Доказательства

A run history cannot show you a run it never created

Источник

Events refused at send are excluded from the numerator and from the denominator, and reported separately.

Источник

Use the Wilson score interval, not the normal approximation.

Источник