Security Benchmark Explorer: An AI Agent That Only Works Because the Content Is Structured

说明与用途

Security Benchmark Explorer利用Sanity的结构化数据，针对54个漏洞检测测试用例（包括5个误报陷阱）提供精确查询。

问题

Security engineers need to query a 54‑case vulnerability benchmark with exact field filters, pattern matching, and aggregations that keyword search cannot provide.

结果

The agent accurately answers complex queries that require exact field matching, pattern matching, and aggregation, demonstrating that structured content is essential for such use cases.

工作流

• 1. User submits a natural‑language query to the chat UI.
• 2. The agent translates the query into a GROQ filter (e.g., caseId == "EVAL-004" or category == "SQL Injection" && difficulty == "hard").
• 3. The agent calls the Sanity Context MCP endpoint with the GROQ query.
• 4. The endpoint returns matching documents as structured JSON.
• 5. The agent reasons over the returned data and generates a concise answer citing case IDs and rationale.
• 6. The answer is displayed to the user in the chat interface.

证据

Security Benchmark Explorer is an AI agent that helps security engineers and AI researchers explore a 54-case benchmark of vulnerability detection test cases — including 5 deliberate false-positive traps designed to distinguish data-flow reasoning from pattern matching.

来源

The agent answers five kinds of questions that a keyword search cannot: Exact case lookup, Compound filtering, Pattern matching on IDs, Field-by-field comparison, Aggregation over structured data.

来源

The agent calls groq_query with a filter, gets structured documents back, and reasons over them. It never guesses — it always queries.

来源

A keyword search for "EVAL-004" returns nothing — the case ID only exists in a structured field.

来源