Security Benchmark Explorer: An AI Agent That Only Works Because the Content Is Structured

Description and use

Security Benchmark Explorer uses structured Sanity data to answer precise queries over a 54‑case vulnerability benchmark, including false‑positive traps.

Problem

Security engineers need to query a 54‑case vulnerability benchmark with exact field filters, pattern matching, and aggregations that keyword search cannot provide.

Outcome

The agent accurately answers complex queries that require exact field matching, pattern matching, and aggregation, demonstrating that structured content is essential for such use cases.

Workflow

• 1. User submits a natural‑language query to the chat UI.
• 2. The agent translates the query into a GROQ filter (e.g., caseId == "EVAL-004" or category == "SQL Injection" && difficulty == "hard").
• 3. The agent calls the Sanity Context MCP endpoint with the GROQ query.
• 4. The endpoint returns matching documents as structured JSON.
• 5. The agent reasons over the returned data and generates a concise answer citing case IDs and rationale.
• 6. The answer is displayed to the user in the chat interface.

Evidence

Security Benchmark Explorer is an AI agent that helps security engineers and AI researchers explore a 54-case benchmark of vulnerability detection test cases — including 5 deliberate false-positive traps designed to distinguish data-flow reasoning from pattern matching.

Source

The agent answers five kinds of questions that a keyword search cannot: Exact case lookup, Compound filtering, Pattern matching on IDs, Field-by-field comparison, Aggregation over structured data.

Source

The agent calls groq_query with a filter, gets structured documents back, and reasons over them. It never guesses — it always queries.

Source

A keyword search for "EVAL-004" returns nothing — the case ID only exists in a structured field.

Source