8 Chinese AI models on the same Pi coding-agent task: what we measured

Description and use

Benchmarking eight Chinese AI models on a single coding task via the Pi agent revealed all models succeeded, offering transparent metrics on requests, tokens, and cost for operational insight.

Problem

Determine how eight Chinese AI models perform on a single coding task when accessed through the same Pi agent, measuring success, request count, token usage, and cost.

Outcome

All eight models succeeded on the task; the benchmark provides transparent metrics on requests, tokens, and cost, enabling operational decision‑making.

Workflow

• 1. Prepare identical fixture and test directory for each model.
• 2. Configure Pi agent with the same provider settings and task contract.
• 3. Run the isolated JavaScript coding task for each model, ensuring separate work directories.
• 4. Record Pi requests, token usage, and production‑audited cost per run.
• 5. Store raw run evidence separately from the task workspace.
• 6. Compile results into a model‑by‑model table and downloadable JSON.
• 7. Publish the benchmark page with methodology notes and data.

Evidence

All eight models completed the task successfully.

Source

45 Pi requests, 94,502 tokens, $0.037618 production‑audited billed cost

Source