My Health Check Watched the Wrong File

Description and use

A health‑check rule that relied on log file age incorrectly flagged live agents as dead because of stdout buffering and infrequent writes. Inspecting state files and script behavior revealed the flaw, leading to a refined approach.

Problem

A health‑check rule that relied on log file age incorrectly flagged live Telegram‑polling agents as dead because the logs were block‑buffered and only written on errors or startup.

Outcome

The author’s initial health‑check rule incorrectly flagged live agents as dead because it relied on log file age, which was affected by stdout buffering and infrequent writes. By inspecting state files and understanding the scripts’ logging behavior, the author identified the flaw and suggested refining the rule to use more reliable artifacts.

Workflow

• 1. Inspect log file mtimes for each agent.
• 2. Apply rule "Artifact age beats PID" to flag jobs with old logs as dead.
• 3. Observe that all three logs had mtime 2026‑08‑11 and mark jobs as dead.
• 4. Realize that the scripts write to stdout only on startup or errors, and stdout is block‑buffered.
• 5. Check state files; notice that state file mtimes are recent, indicating the jobs are alive.
• 6. Analyze script code to understand when logs are written (startup, errors, after handling a message).
• 7. Conclude that log file age is an unreliable signal for these agents.
• 8. Refine health‑check to use state file mtime or other reliable indicators, and consider adding -u to launch agents to avoid buffering.

Evidence

The log had not been touched since August 11 and the state file was seconds old.

Source

An unchanged file does not prove no errors occurred. It proves the file has not been modified.

Source

Both things are true at once: those programs have almost nothing to say, and without -u, the little they do say is not guaranteed to reach disk promptly.

Source