Four times my system looked correct and wasn't

Описание и применение

Автор обнаружил, что тесты, которые казались надёжными, не охватывали критические свойства систем. Путём намеренного нарушения этих свойств он доказал, что тесты пропускают ошибки, и затем исправил их, сделав системы более надёжными.

Задача

Tests passed while the underlying property was still broken in four different systems.

Результат

By intentionally breaking the property under test, the author revealed that existing tests were insufficient. The subsequent fixes made the systems more trustworthy and prevented the same class of bugs.

Рабочий процесс

• 1. Identify a property that matters (e.g., prompt injection safety).
• 2. Write tests that only check superficial aspects.
• 3. Run tests – they pass.
• 4. Break the property intentionally (e.g., use static nonce).
• 5. Observe tests fail, confirming the property is now covered.
• 6. Apply fix (e.g., random nonce, context in queries, allow-listing, CSP tightening).
• 7. Re-run tests – they pass and the property is protected.

Доказательства

The problem: those tests checked that the Python string was assembled correctly. An attacker doesn't care about that.

Источник

Testing against the real client showed that it can send identity attributes, including an email address.

Источник

The quick fix was to loosen the CSP. I removed the incompatible behavior instead, and added CI tests that run against the built HTML, the files that actually ship, checking inline scripts, hashes and style attributes.

Источник