What were my AI-written UI tests actually proving? Green XCUITests over a broken macOS terminal

说明与用途

AI生成的 XCUITest 断言在 Calyx 出现错误时仍通过，揭示了测试动作与真实用户流程之间的差距。通过改写测试以观察守护进程状态并使用正确的启动参数，测试套件现在准确反映用户可见行为并正确清理资源。

问题

AI-generated XCUITest assertions passed while the Calyx terminal app was broken, revealing gaps between test actions and real user interactions.

结果

After refactoring, the test suite no longer passes on broken functionality, accurately reflects user experience, and cleans up leaked shells.

工作流

• Identify failing assertions in persistent session tests
• Move observation from UI to session ledger file
• Remove typeText to avoid keystroke leakage
• Add launch arguments to enable persistent sessions
• Rewrite switch tests to click UI, quit, relaunch, and read back
• Audit all UI tests for what they actually prove
• Replace press(forDuration:thenDragTo:) with click(forDuration:thenDragTo:) for drag events
• Use app.terminate() only for cleanup, not test actions
• Add paired tests to confirm actions occur
• Ensure tests run in sandboxed environment with proper entitlements

证据

The suite was green many times while the product was broken.

来源

The first persistent-session E2E test ran straight into this.

来源

The rewritten test observes the daemon's ledger.

来源

Every persistent-session E2E test still passed, because every one of them turned the feature on with a launch argument before starting the app.

来源