How we validated a from-scratch WebGPU LLM engine, number by number, against llama.cpp

Описание и применение

Валидация WebGPU‑движка LLM с нуля: сравнение токенов, логитов и продолжений с llama.cpp, включая негативные тесты и контроль за включёнными функциями.

Задача

A new inference engine can produce fluent but subtly wrong text; comparing prose is insufficient, so a numeric validation against a reference is required.

Результат

The conformance suite demonstrates that the WebGPU engine matches llama.cpp token‑by‑token and within 2e‑3 on logits for synthetic models across multiple architectures and quantizations, with negative controls that fail when features are disabled.

Рабочий процесс

• Generate tiny random‑weight synthetic models for each architecture using the gguf Python package.
• Quantize each model with llama.cpp (Q2_K–Q6_K, Q8_0) and record token ids, last‑token logits, and greedy continuations.
• Run the tests in headless Chromium with SwiftShader to prove correctness of the WebGPU implementation.
• Compare token ids, logits, and greedy outputs against the reference, ensuring relative logit differences <2e‑3 and identical greedy continuations.
• Run end‑to‑end tests through the HTTP API, chat template, and engine page, checking for text identity and no JS console errors.
• Apply mutation procedures to temporarily disable features (window, norm, bias, etc.) and confirm tests fail, ensuring feature coverage.
• Run automated negative controls (e.g., causal‑only attention in Gemma 3 vision) to detect incorrect implementations.
• Measure performance on multiple machines and document platform pitfalls (GPU watchdog, WebGPU limits, etc.).

Доказательства

A new inference engine has one failure mode that matters more than any crash: it produces fluent text that is subtly wrong.

Источник

We compared numbers, not prose, with llama.cpp as the reference.

Источник

For each prompt the pass criteria are: Token ids identical, Last‑token logits within a relative difference < 2e‑3 of an F32 copy of the same weights, Greedy continuation identical, End‑to‑end through the real HTTP API.

Источник

The engine matches llama.cpp 0.3.35 token for token and within 2e‑3 on logits against dequantized F32, end to end through the HTTP API, with negative controls that fail when they should.

Источник