Your tests passed. Your app didn’t.

Описание и применение

Оптимизация тестов: с помощью Optics можно автоматически обнаруживать скрытые ошибки, отображаемые на экране, добавляя их в отчёт JUnit и заставляя CI падать при наличии ошибок.

Задача

Test suites can pass while the app shows error screens that are not asserted, causing silent failures in CI.

Результат

By defining error messages in a CSV, Optics automatically detects hidden failures, logs them, and can fail CI jobs, ensuring that silent error screens are surfaced without modifying existing tests.

Рабочий процесс

• Add a CSV file named error_definitions.csv to the Optics project with columns error_code,match_string,description,severity.
• Configure Optics to use the file – it is automatically recognized by its columns.
• Run the test suite with "optics execute <suite_name>".
• After the last test case, Optics captures the final screen, searches for each match_string (case‑insensitive) in the visible text, and logs any matches.
• If a match is found, Optics writes a JSON file execution_output/detected_errors_<id>.json, adds a failing test case named on‑screen-error-detection to the JUnit report, and prints a warning.
• Add a CI step after the suite that fails when the JSON file exists, e.g.: - name: Run the suite run: optics execute birch-tests - name: Fail on error screens run: | if compgen -G "birch-tests/execution_output/detected_errors_*.json" > /dev/null; then cat birch-tests/execution_output/detected_errors_*.json exit 1 fi

Доказательства

Optics recognises the file by its error_code and match_string columns, and optics execute uses it on every run.

Источник

After the last test case, Optics reads the final screen, finds “Session expired” and records the match.

Источник

Optics adds each match to the JUnit report as a failing test case named on-screen-error-detection.

Источник