I gave my drift monitor a denominator. The first thing it exposed was a hole in my own data collection
说明与用途
为SEISMOGRAPH添加分母后,发现缺失数据和样本偏差,削弱了漂移检测的可靠性。
问题
The SEISMOGRAPH drift monitor was publishing metrics without denominators, masking data collection gaps and biasing drift detection.
结果
Adding denominators exposed that SEISMOGRAPH was missing data and that surviving samples were biased by provider load, undermining drift detection reliability.
工作流
- 1. Deploy SEISMOGRAPH with a 50‑prompt canary suite scheduled twice daily.
- 2. Add fields to the /v1/weather endpoint to publish sample_count, json_sample_count, length_sample_count, window_start, window_end, and window span.
- 3. Observe the public board and notice that sample_count remains 10 while window spans exceed expected 108 h.
- 4. Reconcile board data with scheduler logs to identify missing runs and failed prompts.
- 5. Analyze run logs to determine that failed prompts caused entire runs to be discarded, leading to biased samples.
- 6. Document findings and publish the updated board and code for community review.
证据
Publishing a rate without its base is a defect; I had been doing it.
The probe runs a 50-prompt canary suite and flushes one aggregated row per run.
The google leg’s ten samples span 176.2 hours: it is emitting at 61% of its own cadence.
- 记录日期: 2026-09-09T03:30:50.941Z