Four Kubernetes node bugs I found by reading CI failures to the end

Описание и применение

Автор выявил четыре ошибки в тестах узлов Kubernetes, обнаружив скрытые проблемы, которые не проявлялись в сообщениях об ошибках, и предложил исправления через pull‑requestы.

Задача

Kubernetes node tests frequently fail, often due to hidden bugs that are not obvious from error messages, leading to flaky CI runs and wasted effort.

Результат

Four bugs were uncovered: a missing image record deletion after GC, a swap test that hung nodes, a memory regression bug, and a hidden bug in kubelet 1.37. Fixes were proposed via pull requests, though none were merged by Oct 10 2026.

Рабочий процесс

• Identify flaky job in TestGrid
• Build a timeline across logs (build, kubelet, serial console)
• Explain differences between failing and passing jobs
• Reproduce failure locally or in unit tests
• Propose and implement fix
• Submit PR for review

Доказательства

Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on.

Источник

The record was written. Then it was deleted one second later. The cached-image theory was wrong.

Источник

The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap.

Источник