What were my AI-written UI tests actually proving? Green XCUITests over a broken macOS terminal

Description and use

AI-generated XCUITest assertions passed while Calyx was broken, exposing gaps between test actions and real user flows. Refactoring tests to observe daemon state and use proper launch arguments made the suite accurately reflect user-visible behavior and clean up resources.

Problem

AI-generated XCUITest assertions passed while the Calyx terminal app was broken, revealing gaps between test actions and real user interactions.

Outcome

After refactoring, the test suite no longer passes on broken functionality, accurately reflects user experience, and cleans up leaked shells.

Workflow

• Identify failing assertions in persistent session tests
• Move observation from UI to session ledger file
• Remove typeText to avoid keystroke leakage
• Add launch arguments to enable persistent sessions
• Rewrite switch tests to click UI, quit, relaunch, and read back
• Audit all UI tests for what they actually prove
• Replace press(forDuration:thenDragTo:) with click(forDuration:thenDragTo:) for drag events
• Use app.terminate() only for cleanup, not test actions
• Add paired tests to confirm actions occur
• Ensure tests run in sandboxed environment with proper entitlements

Evidence

The suite was green many times while the product was broken.

Source

The first persistent-session E2E test ran straight into this.

Source

The rewritten test observes the daemon's ledger.

Source

Every persistent-session E2E test still passed, because every one of them turned the feature on with a launch argument before starting the app.

Source