Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

374 tests went in, 352 came out, and the build was green

Описание и применение

При запуске тестов winwright 374 теста, затем 352, сборка осталась зеленой, но 22 теста пропали из‑за падения хоста, что привело к ложному положительному результату.

Задача

A test suite reported a green build after a reduction in test count, but 22 tests disappeared, leading to a false sense of correctness.

Результат

The build was green despite 22 tests missing, illustrating the need for a third test outcome (hole) to detect incomplete runs.

Рабочий процесс

  • Run winwright test suite
  • Observe test count and exit code
  • Identify missing tests due to host crash
  • Recognize exit code 0 as green but not all tests executed

Доказательства

Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's.

Источник

Every tool in the chain did what it was designed to do. The result was still false.

Источник

winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing.

Источник

  • Дата записи: 2026-10-13T15:41:31.419Z