What were my AI-written UI tests actually proving? Green XCUITests over a broken macOS terminal

Описание и применение

Автоматизированные тесты XCUITest, генерируемые агентами, проходили, несмотря на ошибки в приложении Calyx, выявив разрывы между действиями тестов и реальными пользовательскими сценариями. Переписав тесты, чтобы наблюдать за состоянием демона и использовать корректные пути запуска, тесты теперь отражают видимое пользователю поведение и корректно очищают ресурсы.

Задача

AI-generated XCUITest assertions passed while the Calyx terminal app was broken, revealing gaps between test actions and real user interactions.

Результат

After refactoring, the test suite no longer passes on broken functionality, accurately reflects user experience, and cleans up leaked shells.

Рабочий процесс

• Identify failing assertions in persistent session tests
• Move observation from UI to session ledger file
• Remove typeText to avoid keystroke leakage
• Add launch arguments to enable persistent sessions
• Rewrite switch tests to click UI, quit, relaunch, and read back
• Audit all UI tests for what they actually prove
• Replace press(forDuration:thenDragTo:) with click(forDuration:thenDragTo:) for drag events
• Use app.terminate() only for cleanup, not test actions
• Add paired tests to confirm actions occur
• Ensure tests run in sandboxed environment with proper entitlements

Доказательства

The suite was green many times while the product was broken.

Источник

The first persistent-session E2E test ran straight into this.

Источник

The rewritten test observes the daemon's ledger.

Источник

Every persistent-session E2E test still passed, because every one of them turned the feature on with a launch argument before starting the app.

Источник