Integrating Mutation Testing (PITest) to Detect Insufficient Selenium Test Assertions

Описание и применение

Интеграция PITest позволяет выявить слабые утверждения в Selenium‑тестах, даже при 100% покрытии кода, и повысить их надёжность.

Задача

High code coverage in Selenium often creates a false sense of security. Tests can execute every line of a Page Object or utility but fail to verify the actual outcome, leading to silent failures.

Результат

Adding assertions and interaction verifications turned surviving mutants into killed mutants, proving the tests now validate actual outcomes.

Рабочий процесс

• 1. Run Selenium tests with 100% coverage.
• 2. Execute PITest mutation testing on the utility classes.
• 3. Observe surviving mutants indicating weak assertions.
• 4. Refactor tests to include assertions on return values and verify method interactions.
• 5. Re-run PITest to confirm mutants are killed.

Доказательства

High code coverage in Selenium often creates a false sense of security. Tests can execute every line of a Page Object or utility but fail to verify the actual outcome (e.g., data is correct, element state changed).

Источник

PITest Result: SURVIVED mutants! Return Value Mutant: return wait.until(...); → return null; Test passed as null was ignored.

Источник

With these assertions, PITest will report "KILLED" mutants, validating the test's effectiveness.

Источник