800 Million Tokens for $36: Benchmarking DSH + DeepSeek V4 Pro on a Real Codebase

Description and use

The DeepSeek Harness benchmark demonstrated that an LLM can sustain real‑world development, making 45 commits over several days, processing 612 million tokens for $28.35, and delivering significant architectural, algorithmic, and formal verification improvements.

Problem

Most coding‑agent benchmarks evaluate isolated fixes, not sustained, real‑world development.

Outcome

The agent performed 45 commits, processed 612 M tokens for $28.35, and produced substantial architectural changes, algorithmic improvements, and formal proofs, demonstrating sustained, iterative engineering behavior.

Workflow

• Set up DSH with DeepSeek V4 Pro
• Ran the agent on the repo for several days
• Collected API usage and token metrics
• Reviewed commits and diffs
• Analyzed algorithmic changes and benchmarks
• Documented outcomes and lessons

Evidence

612,923,681 tokens processed for $28.35

Source

45 commits across 193 changed files

Source

Rewrite Natural storage around 64-bit limbs

Source