Four Kubernetes node bugs I found by reading CI failures to the end

Description and use

The author uncovered four Kubernetes node bugs by deeply analyzing CI failures, revealing hidden issues and proposing fixes via pull requests.

Problem

Kubernetes node tests frequently fail, often due to hidden bugs that are not obvious from error messages, leading to flaky CI runs and wasted effort.

Outcome

Four bugs were uncovered: a missing image record deletion after GC, a swap test that hung nodes, a memory regression bug, and a hidden bug in kubelet 1.37. Fixes were proposed via pull requests, though none were merged by Oct 10 2026.

Workflow

• Identify flaky job in TestGrid
• Build a timeline across logs (build, kubelet, serial console)
• Explain differences between failing and passing jobs
• Reproduce failure locally or in unit tests
• Propose and implement fix
• Submit PR for review

Evidence

Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on.

Source

The record was written. Then it was deleted one second later. The cached-image theory was wrong.

Source

The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap.

Source