My Health Check Watched the Wrong File

Description and use

A health‑check rule that used log file age incorrectly flagged live Telegram polling scripts as dead because stdout was block‑buffered and logs were only written on startup or errors. State file mtimes provided a more accurate signal of activity.

Problem

A health‑check rule that relied on log file modification time incorrectly flagged live Telegram‑poller jobs as dead because the logs were block‑buffered and only written on errors or startup.

Outcome

The health‑check rule was too simplistic; log file age alone is insufficient when stdout is block‑buffered and logs are only written on errors or startup. State file mtimes and contents are more reliable indicators of activity for these polling scripts.

Workflow

• 1. Apply the rule to the machine and observe false positives.
• 2. Inspect the log files and note they are unchanged.
• 3. Examine the scripts to see when they write to stdout and to state files.
• 4. Realise that stdout is block‑buffered unless -u is used.
• 5. Check state file mtimes and contents to confirm activity.
• 6. Conclude that the rule must be refined to consider the specific artifact and its write semantics.

Evidence

A job that is loaded, looks healthy to the supervisor, and is producing nothing is the failure worth catching.

Source

The log had not been touched since August 11 and the state file was seconds old.

Source

An unchanged file does not prove no errors occurred. It proves the file has not been modified.

Source

Those three logs will still carry mtime 2026-08-11 on 2026-10-01, while the processes are still alive.

Source