374 tests went in, 352 came out, and the build was green

Description and use

Running winwright with 374 tests, then 352, the build stayed green but 22 tests disappeared due to host crash, causing a false positive.

Problem

A test suite reported a green build after a reduction in test count, but 22 tests disappeared, leading to a false sense of correctness.

Outcome

The build was green despite 22 tests missing, illustrating the need for a third test outcome (hole) to detect incomplete runs.

Workflow

• Run winwright test suite
• Observe test count and exit code
• Identify missing tests due to host crash
• Recognize exit code 0 as green but not all tests executed

Evidence

Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's.

Source

Every tool in the chain did what it was designed to do. The result was still false.

Source

winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing.

Source