Security Benchmark Explorer: An AI Agent That Only Works Because the Content Is Structured

Описание и применение

Агент Security Benchmark Explorer использует структурированные данные Sanity для точных запросов к 54‑кратному набору тестов на уязвимости, включая ложные положительные ловушки.

Задача

Security engineers need to query a 54‑case vulnerability benchmark with exact field filters, pattern matching, and aggregations that keyword search cannot provide.

Результат

The agent accurately answers complex queries that require exact field matching, pattern matching, and aggregation, demonstrating that structured content is essential for such use cases.

Рабочий процесс

• 1. User submits a natural‑language query to the chat UI.
• 2. The agent translates the query into a GROQ filter (e.g., caseId == "EVAL-004" or category == "SQL Injection" && difficulty == "hard").
• 3. The agent calls the Sanity Context MCP endpoint with the GROQ query.
• 4. The endpoint returns matching documents as structured JSON.
• 5. The agent reasons over the returned data and generates a concise answer citing case IDs and rationale.
• 6. The answer is displayed to the user in the chat interface.

Доказательства

Security Benchmark Explorer is an AI agent that helps security engineers and AI researchers explore a 54-case benchmark of vulnerability detection test cases — including 5 deliberate false-positive traps designed to distinguish data-flow reasoning from pattern matching.

Источник

The agent answers five kinds of questions that a keyword search cannot: Exact case lookup, Compound filtering, Pattern matching on IDs, Field-by-field comparison, Aggregation over structured data.

Источник

The agent calls groq_query with a filter, gets structured documents back, and reasons over them. It never guesses — it always queries.

Источник

A keyword search for "EVAL-004" returns nothing — the case ID only exists in a structured field.

Источник