Four Kubernetes node bugs I found by reading CI failures to the end

说明与用途

作者通过深入分析 CI 失败，发现了四个 Kubernetes 节点错误，并通过拉取请求提出了解决方案。

问题

Kubernetes node tests frequently fail, often due to hidden bugs that are not obvious from error messages, leading to flaky CI runs and wasted effort.

结果

Four bugs were uncovered: a missing image record deletion after GC, a swap test that hung nodes, a memory regression bug, and a hidden bug in kubelet 1.37. Fixes were proposed via pull requests, though none were merged by Oct 10 2026.

工作流

• Identify flaky job in TestGrid
• Build a timeline across logs (build, kubelet, serial console)
• Explain differences between failing and passing jobs
• Reproduce failure locally or in unit tests
• Propose and implement fix
• Submit PR for review

证据

Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on.

来源

The record was written. Then it was deleted one second later. The cached-image theory was wrong.

来源

The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap.

来源