My monitoring tool lied to me and secretly deleted all past data integrity issues

Description and use

The monitoring script mistakenly cleared all unresolved data gaps, hiding them. The fix added a scan range check to prevent deletion of historical records.

Problem

A custom monitoring script silently cleared unresolved data gaps, causing loss of historical issue records.

Outcome

The fix prevented silent deletion of unresolved data gaps, ensuring accurate monitoring and alerting.

Workflow

• Scan each bot’s data store to list missing dates.
• Load unresolved_issues.json.
• Compare current missing dates with unresolved issues.
• If new missing dates exist, notify Slack and add to JSON.
• If a previously unresolved issue is not in current missing dates, check expected_first; if the issue date is newer than expected_first, consider it resolved; otherwise keep it.
• Save updated unresolved_issues.json.
• Run the script continuously.
• Monitor Slack notifications for anomalies.
• Implement fail‑safe logic to avoid clearing unresolved issues when scan range is uncertain.

Evidence

If an issue is in unresolved_issues.json but not found in the current scan, assume it's "resolved" and remove it from the list.

Source

The monitoring tool now checks this expected_first. Only if an unresolved issue date in unresolved_issues.json is newer than expected_first does it consider it "resolved."

Source