374 tests went in, 352 came out, and the build was green
说明与用途
运行 winwright 374 个测试后,随后 352 个测试,构建保持绿色,但由于主机崩溃,22 个测试消失,导致错误的正面结果。
问题
A test suite reported a green build after a reduction in test count, but 22 tests disappeared, leading to a false sense of correctness.
结果
The build was green despite 22 tests missing, illustrating the need for a third test outcome (hole) to detect incomplete runs.
工作流
- Run winwright test suite
- Observe test count and exit code
- Identify missing tests due to host crash
- Recognize exit code 0 as green but not all tests executed
证据
Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's.
Every tool in the chain did what it was designed to do. The result was still false.
winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing.
- 记录日期: 2026-10-13T15:41:31.419Z