SerpApi Achieves 97.6% Accuracy on SimpleQA Verified

Описание и применение

SerpApi в сочетании с GPT‑6.1 Sol достиг 97,6 % точности на наборе SimpleQA Verified, используя Google Search API с Markdown‑выводом и Extract API для извлечения контента страниц.

Задача

How to evaluate AI agents’ ability to answer factual questions using web search and extraction APIs.

Результат

SerpApi’s combined search and extraction approach achieved 97.6% accuracy on SimpleQA Verified, matching Tavily’s score and topping other providers’ reported accuracies.

Рабочий процесс

• 1. Connect SerpApi Google Search API (engine=google) with Markdown output (output=md) to GPT‑6.1 Sol.
• 2. Use SerpApi Extract API (engine=extract, mode=scrape, include_markdown=true, json_restrictor=text_content.markdown) to fetch page content when needed.
• 3. Allow multiple searches and fetches per question; limit each page fetch to 48,000 characters and each model input to 56,000 tokens.
• 4. Let the model decide when to search or read a page within these limits.
• 5. Grade final answers with GPT‑5.6 Sol (high reasoning) without tool access.
• 6. Exclude any responses where search results exposed benchmark answers or hints (40 questions).

Доказательства

We excluded 40 questions where we detected exposure to public benchmark answers or guidance.

Источник

The model answered 937 of the remaining 960 questions correctly, giving us 97.6% accuracy.

Источник

We used the regular SerpApi Google Search API with Markdown output.

Источник

We used engine=extract with mode=scrape and returned the page's Markdown through include_markdown=true and json_restrictor=text_content.markdown.

Источник