Four times my system looked correct and wasn't

Description and use

The author showed that seemingly passing tests can miss critical properties. By breaking those properties intentionally, he revealed hidden bugs and fixed them, improving system reliability.

Problem

Tests passed while the underlying property was still broken in four different systems.

Outcome

By intentionally breaking the property under test, the author revealed that existing tests were insufficient. The subsequent fixes made the systems more trustworthy and prevented the same class of bugs.

Workflow

• 1. Identify a property that matters (e.g., prompt injection safety).
• 2. Write tests that only check superficial aspects.
• 3. Run tests – they pass.
• 4. Break the property intentionally (e.g., use static nonce).
• 5. Observe tests fail, confirming the property is now covered.
• 6. Apply fix (e.g., random nonce, context in queries, allow-listing, CSP tightening).
• 7. Re-run tests – they pass and the property is protected.

Evidence

The problem: those tests checked that the Python string was assembled correctly. An attacker doesn't care about that.

Source

Testing against the real client showed that it can send identity attributes, including an email address.

Source

The quick fix was to loosen the CSP. I removed the incompatible behavior instead, and added CI tests that run against the built HTML, the files that actually ship, checking inline scripts, hashes and style attributes.

Source