I said my automation never ran. Five file timestamps proved me wrong.

Description and use

The job ran, but a stale status file misled the operator. Adding checks on file timestamps and logs prevented false negatives.

Problem

Assuming a scheduled job had stopped because a state file was stale, leading to unnecessary debugging and client communication.

Outcome

Automation ran correctly; the stale state file misled the operator. Implementing multiple evidence checks prevented unnecessary debugging.

Workflow

• 1. Check state file timestamp. 2. Inspect queue file modification times. 3. Verify log file existence for the date. 4. Compare status field with other evidence. 5. Update monitoring to require all three evidence classes before flagging failure.
• 6. Adjust job to write status only after successful completion.
• 7. Investigate and fix quota truncation issue.

Evidence

I now require three evidence classes before concluding a job did not run. Queue file modification times. Whether a log file exists for that date. And only then the status field.

Source

The automation had run. Five times. It just never wrote back the status field I was using as my only signal.

Source

If a job is interrupted, or crashes after doing its real work, or writes status only on certain paths, the status field lies.

Source