My monitoring tool lied to me and secretly deleted all past data integrity issues

Описание и применение

Скрипт мониторинга автоматически удалил все незавершенные проблемы с данными, скрывая их. Исправление добавило проверку диапазона сканирования, чтобы не удалять исторические записи.

Задача

A custom monitoring script silently cleared unresolved data gaps, causing loss of historical issue records.

Результат

The fix prevented silent deletion of unresolved data gaps, ensuring accurate monitoring and alerting.

Рабочий процесс

• Scan each bot’s data store to list missing dates.
• Load unresolved_issues.json.
• Compare current missing dates with unresolved issues.
• If new missing dates exist, notify Slack and add to JSON.
• If a previously unresolved issue is not in current missing dates, check expected_first; if the issue date is newer than expected_first, consider it resolved; otherwise keep it.
• Save updated unresolved_issues.json.
• Run the script continuously.
• Monitor Slack notifications for anomalies.
• Implement fail‑safe logic to avoid clearing unresolved issues when scan range is uncertain.

Доказательства

If an issue is in unresolved_issues.json but not found in the current scan, assume it's "resolved" and remove it from the list.

Источник

The monitoring tool now checks this expected_first. Only if an unresolved issue date in unresolved_issues.json is newer than expected_first does it consider it "resolved."

Источник