Your tests passed. Your app didn’t.

Description and use

Optimize tests: with Optics you can automatically detect hidden on‑screen errors, add them to the JUnit report, and make CI fail when errors are present.

Problem

Test suites can pass while the app shows error screens that are not asserted, causing silent failures in CI.

Outcome

By defining error messages in a CSV, Optics automatically detects hidden failures, logs them, and can fail CI jobs, ensuring that silent error screens are surfaced without modifying existing tests.

Workflow

• Add a CSV file named error_definitions.csv to the Optics project with columns error_code,match_string,description,severity.
• Configure Optics to use the file – it is automatically recognized by its columns.
• Run the test suite with "optics execute <suite_name>".
• After the last test case, Optics captures the final screen, searches for each match_string (case‑insensitive) in the visible text, and logs any matches.
• If a match is found, Optics writes a JSON file execution_output/detected_errors_<id>.json, adds a failing test case named on‑screen-error-detection to the JUnit report, and prints a warning.
• Add a CI step after the suite that fails when the JSON file exists, e.g.: - name: Run the suite run: optics execute birch-tests - name: Fail on error screens run: | if compgen -G "birch-tests/execution_output/detected_errors_*.json" > /dev/null; then cat birch-tests/execution_output/detected_errors_*.json exit 1 fi

Evidence

Optics recognises the file by its error_code and match_string columns, and optics execute uses it on every run.

Source

After the last test case, Optics reads the final screen, finds “Session expired” and records the match.

Source

Optics adds each match to the JUnit report as a failing test case named on-screen-error-detection.

Source