Four times my system looked correct and wasn't
说明与用途
作者证明了看似通过的测试可能忽略关键属性。通过有意破坏这些属性,他发现隐藏的错误并修复,提升了系统可靠性。
问题
Tests passed while the underlying property was still broken in four different systems.
结果
By intentionally breaking the property under test, the author revealed that existing tests were insufficient. The subsequent fixes made the systems more trustworthy and prevented the same class of bugs.
工作流
- 1. Identify a property that matters (e.g., prompt injection safety).
- 2. Write tests that only check superficial aspects.
- 3. Run tests – they pass.
- 4. Break the property intentionally (e.g., use static nonce).
- 5. Observe tests fail, confirming the property is now covered.
- 6. Apply fix (e.g., random nonce, context in queries, allow-listing, CSP tightening).
- 7. Re-run tests – they pass and the property is protected.
证据
The problem: those tests checked that the Python string was assembled correctly. An attacker doesn't care about that.
Testing against the real client showed that it can send identity attributes, including an email address.
The quick fix was to loosen the CSP. I removed the incompatible behavior instead, and added CI tests that run against the built HTML, the files that actually ship, checking inline scripts, hashes and style attributes.
- 记录日期: 2026-10-26T02:11:33.450Z