The gap between scheduled and started was free telemetry we kept discarding

Description and use

Recording actual start times in unattended job logs exposed drift, enabled distinguishing late from missing runs, removed implicit timing dependencies, and correctly attributed delays to scheduler contention.

Problem

Unattended scheduled jobs lacked visibility into start‑time drift, making it impossible to distinguish late starts from failures and causing hidden dependencies and misattributed faults.

Outcome

Adding a single timestamp field revealed start‑time drift, improved failure detection, eliminated hidden dependencies, and shifted fault attribution away from the agent.

Workflow

• 1. Modify job to record intended start time and actual start time. 2. Store both timestamps in the job log. 3. Use the difference to flag late starts. 4. Replace implicit ordering logic with explicit completion signals. 5. Adjust tolerance or reschedule to mitigate scheduler contention.

Evidence

We could tell late from never. Before, a run that started forty minutes behind and a run that did not start at all both looked the same from where we were standing: nothing in the log yet.

Source

Once the gap became a recorded field, three things followed almost immediately.

Source