Claude, Codex, Kimi K3, Qwen: I graded their web search 120 times

Description and use

The study compared web‑search accuracy across five setups: Claude, OpenAI live, OpenAI cached, Kimi K3, and local Qwen. OpenAI cached search achieved perfect accuracy (24/24), live search 22/24, Claude and Kimi K3 21/24, and local Qwen 9/24.

Problem

Determine which vendor’s web search yields the most accurate answers for a set of 8 questions, including recent and language‑specific queries.

Outcome

OpenAI’s cached search produced the highest accuracy (100%) and fastest median time (6.2 s). Live search was close (22/24). Claude and Kimi K3 matched cached accuracy (21/24) but were slower. Local Qwen lagged (9/24). The same search API yielded different results depending on the model’s reading capability.

Workflow

• Define 8 test questions with verifiable answers
• Set up five configurations: Claude, OpenAI live, OpenAI cached, Kimi K3, local Qwen
• Run each configuration 3 times for each question (120 total)
• Collect and grade answers for correctness, partial, or wrong
• Tabulate results and analyze performance per setup

Evidence

Only cached search got every answer right.

Source

The other surprise: the same search API produced 21 correct answers with one model and 9 with another.

Source

OpenAI defaults to cached; --search for live

Source