374 tests went in, 352 came out, and the build was green
Описание и применение
При запуске тестов winwright 374 теста, затем 352, сборка осталась зеленой, но 22 теста пропали из‑за падения хоста, что привело к ложному положительному результату.
Задача
A test suite reported a green build after a reduction in test count, but 22 tests disappeared, leading to a false sense of correctness.
Результат
The build was green despite 22 tests missing, illustrating the need for a third test outcome (hole) to detect incomplete runs.
Рабочий процесс
- Run winwright test suite
- Observe test count and exit code
- Identify missing tests due to host crash
- Recognize exit code 0 as green but not all tests executed
Доказательства
Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's.
Every tool in the chain did what it was designed to do. The result was still false.
winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing.
- Дата записи: 2026-10-13T15:41:31.419Z