Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I gave my drift monitor a denominator. The first thing it exposed was a hole in my own data collection

Description and use

Adding denominators to SEISMOGRAPH exposed missing data and biased sampling, undermining drift detection reliability.

Problem

The SEISMOGRAPH drift monitor was publishing metrics without denominators, masking data collection gaps and biasing drift detection.

Outcome

Adding denominators exposed that SEISMOGRAPH was missing data and that surviving samples were biased by provider load, undermining drift detection reliability.

Workflow

  • 1. Deploy SEISMOGRAPH with a 50‑prompt canary suite scheduled twice daily.
  • 2. Add fields to the /v1/weather endpoint to publish sample_count, json_sample_count, length_sample_count, window_start, window_end, and window span.
  • 3. Observe the public board and notice that sample_count remains 10 while window spans exceed expected 108 h.
  • 4. Reconcile board data with scheduler logs to identify missing runs and failed prompts.
  • 5. Analyze run logs to determine that failed prompts caused entire runs to be discarded, leading to biased samples.
  • 6. Document findings and publish the updated board and code for community review.

Evidence

Publishing a rate without its base is a defect; I had been doing it.

Source

The probe runs a 50-prompt canary suite and flushes one aggregated row per run.

Source

The google leg’s ten samples span 176.2 hours: it is emitting at 61% of its own cadence.

Source

  • Record date: 2026-09-09T03:30:50.941Z