A Government Lab Spent £1,000 and Two Weeks Using Frontier AI to Find Cloud Misconfigurations. The Same Findings Cost $0.

说明与用途

政府实验室使用 frontier AI 在 AWS 中发现配置错误，花费 £1,000 并耗时两周，但同样的错误可以通过 Stave 的确定性评估免费且即时发现。

问题

Determine whether frontier AI is cost‑effective for finding cloud misconfigurations compared to deterministic configuration evaluation.

结果

Frontier AI found a 5‑step privilege‑escalation chain for £150–1,000 and two weeks, but the same chain was deterministically discoverable for $0 in seconds using Stave.

工作流

• 1. Selected three approaches: source‑code analysis, agentic probing, human‑in‑the‑loop red‑teaming.
• 2. Ran each approach over two weeks, consuming £150–1,000 in LLM tokens.
• 3. Triaged findings, separating noise from actionable issues.
• 4. Compared results to deterministic configuration evaluation.
• 5. Concluded that deterministic evaluation is cheaper, faster, and reproducible for configuration‑based findings.
• 6. Recommended using AI for application‑logic bugs and creative attack surface exploration.
• 7. Suggested continuous deterministic evaluation for infrastructure misconfigurations.

证据

"The cost was under £1,000 in LLM tokens. The most capable model found the critical chain for under £150."

来源

"The 5-step attack chain was the exercise's most significant discovery."

来源

"A deterministic configuration evaluator takes a snapshot of the cloud infrastructure, including the IAM policies, resource configurations, trust relationships, organizational policies to evaluate compound predicates against it."

来源

"The cost difference is categorical. One approach consumes frontier model compute for every exercise. The other runs locally, offline, with no external dependencies, on every commit."

来源