Rescuing Deleted Git Commits from Automated Nightly Runners

Description and use

The author describes an incident where an automated runner deleted local branches containing unpushed commits after reporting a push failure. Recovery was achieved using git fsck. The script was fixed to verify commits by comparing HEAD, push from the parent repo, and only delete branches after verified remote success.

Problem

An automated nightly runner reported push failures and requested manual intervention, but had already deleted the local branches and worktrees containing the unpushed commits, effectively destroying the work it claimed needed saving.

Outcome

Four repositories were successfully recovered using git fsck. The automation script was patched to prevent future data loss by verifying state directly rather than relying on error messages, and by ensuring destructive cleanup only happens after verified remote success.

Workflow

• Identify dangling commits using git fsck --no-reflogs --lost-found.
• Inspect the dangling commit hash using git log -1 --oneline <hash> to verify it is the lost work.
• Create a new branch pointing to the dangling commit using git branch rescue/<date> <hash>.
• Fix the automation script to check for commits by comparing HEAD before and after the child process runs.
• Fix the automation script to push from the parent checkout instead of the worktree.
• Fix the automation script to only delete the worktree/branch if the remote has all commits (unpushed count is 0).

Evidence

The rescue is three commands... git fsck --no-reflogs --lost-found... git branch rescue/2026-08-21 3f9c2a1

Source

The destructive step in any automation has to be conditioned on the verified success, not the reported one.

Source