Claude, Codex, Kimi K3, Qwen: I graded their web search 120 times

说明与用途

本实验比较了五种配置的网页搜索准确性：Claude、OpenAI（实时和缓存）、Kimi K3 和本地 Qwen。OpenAI 缓存搜索实现了 100% 准确率（24/24），实时搜索 22/24，Claude 和 Kimi K3 21/24， 本地 Qwen 9/24。

问题

Determine which vendor’s web search yields the most accurate answers for a set of 8 questions, including recent and language‑specific queries.

结果

OpenAI’s cached search produced the highest accuracy (100%) and fastest median time (6.2 s). Live search was close (22/24). Claude and Kimi K3 matched cached accuracy (21/24) but were slower. Local Qwen lagged (9/24). The same search API yielded different results depending on the model’s reading capability.

工作流

• Define 8 test questions with verifiable answers
• Set up five configurations: Claude, OpenAI live, OpenAI cached, Kimi K3, local Qwen
• Run each configuration 3 times for each question (120 total)
• Collect and grade answers for correctness, partial, or wrong
• Tabulate results and analyze performance per setup

证据

Only cached search got every answer right.

来源

The other surprise: the same search API produced 21 correct answers with one model and 9 with another.

来源

OpenAI defaults to cached; --search for live

来源