Benchmarking 6 Open LLMs on a 2-Machine RPC Cluster (Consumer GPUs, ~96GB VRAM)

Description and use

Qwen3.6-35B-A3B Q8_0 was the best-performing model overall: 88 tok/s decode TPS, 128K max usable context, 35.6% translation error rate, 83% code understanding, 18.8% hallucination rate, and 100% proofreading. Llama-3.3-70B Q6_K ranked last due to low speed (~10 tok/s) and high error rate (54.5%). gpt-oss-120b MXFP4 had the highest hallucination rate (56.3%) despite fastest decode (94 tok/s).

Problem

Evaluate and rank six open-source LLMs across speed, translation error rate, code understanding, hallucination rate, and proofreading quality using a dual-node RPC cluster with consumer GPUs totaling ~96GB VRAM.

Outcome

Qwen3.6-35B-A3B Q8_0 was the best-performing model overall: 88 tok/s decode TPS, 128K max usable context, 35.6% translation error rate, 83% code understanding, 18.8% hallucination rate, and 100% proofreading. Llama-3.3-70B Q6_K ranked last due to low speed (~10 tok/s) and high error rate (54.5%). gpt-oss-120b MXFP4 had the highest hallucination rate (56.3%) despite fastest decode (94 tok/s).

Workflow

• Connect two machines with an Ethernet cable (CAT6A, ~5Gbps nominal).
• Install llama.cpp and configure the RPC server on both hosts.
• Set --tensor-split to allocate the largest share to the main card (initially set to 32,16,16,16,16).
• Load each candidate model and measure VRAM usage per model across Host 1 (RTX 5080 + RTX 5060 Ti×2) and Host 2 (RTX 5090 + RTX 5060 Ti).
• Run automated load tests for Speed / Concurrency (TTFT, decode TPS, degradation at 1/2/4 concurrency).
• Run translation error rate tests (101 cases, JP to ZH) using real novel corpora including classic mistranslation traps.
• Run code understanding tests (6 high-difficulty cases: bug troubleshooting, behavior prediction, hierarchical allocator, TP bandwidth reasoning, bash traps, refactoring).
• Run hallucination rate tests (24 cases covering false premises, fake citations, fabricated terms, unknowable questions, translation traps).
• Run translation proofreading/correction tests (9 cases scored separately from scratch translation).
• Judge results primarily with Claude 3.5 Sonnet (Opus 4.8 equivalent), supplemented by partial manual review.
• Adjust tensor-split order based on observed VRAM allocation (remote RPC cards placed before local main card).
• Compare final rankings across all five dimensions.

Evidence

In the endless pursuit of 'bigger, bigger, and even bigger,' I finally reached this point: two machines directly connected via a local area network, using RPC scheduling to split and load the model layer by layer across both hosts.

Source

I have a limited budget. Buying a professional-grade graphics card with 96GB of VRAM would definitely not pass my wife's approval. So for consumer-level GPU users on a budget like me, linking multiple machines and GPUs via RPC is currently the only realistic option.

Source

Under this bandwidth, llama.cpp's RPC chain invocation is currently the only viable choice (a Thunderbolt 5 adapter is on the way; once it arrives, it should close some of the performance gap).

Source

Once the two machines were connected, it was time to pick the models. With a 9