Claude, Codex, Kimi K3, Qwen: I graded their web search 120 times

Описание и применение

Тест сравнил точность веб‑поиска в пяти конфигурациях: Claude, OpenAI (живой и кэшированный), Kimi K3 и локальный Qwen. Кэшированный поиск OpenAI показал 100 % точности, живой – 92 %. Kimi K3 и Claude достигли 88 %, а локальный Qwen – 38 %.

Задача

Determine which vendor’s web search yields the most accurate answers for a set of 8 questions, including recent and language‑specific queries.

Результат

OpenAI’s cached search produced the highest accuracy (100%) and fastest median time (6.2 s). Live search was close (22/24). Claude and Kimi K3 matched cached accuracy (21/24) but were slower. Local Qwen lagged (9/24). The same search API yielded different results depending on the model’s reading capability.

Рабочий процесс

• Define 8 test questions with verifiable answers
• Set up five configurations: Claude, OpenAI live, OpenAI cached, Kimi K3, local Qwen
• Run each configuration 3 times for each question (120 total)
• Collect and grade answers for correctness, partial, or wrong
• Tabulate results and analyze performance per setup

Доказательства

Only cached search got every answer right.

Источник

The other surprise: the same search API produced 21 correct answers with one model and 9 with another.

Источник

OpenAI defaults to cached; --search for live

Источник