I gave my drift monitor a denominator. The first thing it exposed was a hole in my own data collection
Описание и применение
Добавление знаменателя в SEISMOGRAPH выявило пропущенные данные и смещение выборки, что подорвало надёжность обнаружения дрейфа.
Задача
The SEISMOGRAPH drift monitor was publishing metrics without denominators, masking data collection gaps and biasing drift detection.
Результат
Adding denominators exposed that SEISMOGRAPH was missing data and that surviving samples were biased by provider load, undermining drift detection reliability.
Рабочий процесс
- 1. Deploy SEISMOGRAPH with a 50‑prompt canary suite scheduled twice daily.
- 2. Add fields to the /v1/weather endpoint to publish sample_count, json_sample_count, length_sample_count, window_start, window_end, and window span.
- 3. Observe the public board and notice that sample_count remains 10 while window spans exceed expected 108 h.
- 4. Reconcile board data with scheduler logs to identify missing runs and failed prompts.
- 5. Analyze run logs to determine that failed prompts caused entire runs to be discarded, leading to biased samples.
- 6. Document findings and publish the updated board and code for community review.
Доказательства
Publishing a rate without its base is a defect; I had been doing it.
The probe runs a 50-prompt canary suite and flushes one aggregated row per run.
The google leg’s ten samples span 176.2 hours: it is emitting at 61% of its own cadence.
- Дата записи: 2026-09-09T03:30:50.941Z