Harvey's Margin Went to -50%. Cheaper Tokens Didn't Fix It — Better Data Did

Description and use

Margin dropped due to token cost spike; solved by switching to an open-weight model and domain-specific post-training, cutting cost per task 10x.

Problem

Agentic AI usage caused token consumption to spike 20x, turning a +50% gross margin into -50% despite revenue growth.

Outcome

Switching to a cheaper open-weight model alone was insufficient; domain-specific post-training and rigorous evaluation reduced cost per completed task by 10x and restored profitability.

Workflow

• 1. Identify token cost spike due to agentic usage.
• 2. Switch to cheaper open-weight model (Moonshot AI Kimi K3 post-trained with Fireworks AI).
• 3. Collect domain-specific post-training data (SFT, RLHF, preference ranking, correction).
• 4. Build parity benchmark against production task distribution.
• 5. Evaluate new model for hallucinations, error modes, and performance parity.
• 6. Deploy new model with monitoring for drift and ongoing QA.
• 7. Iterate post-training and evaluation to maintain cost and quality.

Evidence

Harvey reported roughly a 10x reduction in cost per cell on its Review Tables product and a 90% reduction in cost per query on Firm Knowledge.

Source

The model swap is the easy 10% of the project. The post-training data and the evaluation harness that proves the new model is actually as good is the other 90%.

Source