The fine‑tune that looked fine

Description and use

Fine‑tuning a local model did not match Claude’s decision quality for job‑description triage: while rank correlation was similar, the fine‑tuned model recalled fewer strong roles and did not reduce false alarms. The original model remains in production.

Problem

Can a fine‑tuned local AI model replace an expensive Claude model for job‑description triage while matching its decision quality?

Outcome

Fine‑tuning reduced prompt length but failed to match Claude on strong‑role recall; the original model remains in production.

Workflow

• Collect 1,160 Claude evaluations (job description, score, reasoning).
• Hold out 116 evaluations (14 strong) for testing.
• Train three fine‑tuned models on the remaining 1,044: V1 repeats strong roles 3×, V2 repeats strong roles 6× and near‑strong 2×, V3 trains on scores only.
• Evaluate each model on the held‑out set, measuring rank correlation, strong‑role recall, false alarms, and big misses.
• Analyze score distributions and attempt post‑processing adjustments (shift, stretch, linear fit, cutoff change).
• Conclude that fine‑tuning did not improve strong‑role recall and the original model remains in production.

Evidence

My fine‑tuned AI model matched the original on the headline metric and cut false alarms by more than half. It also found 6 of the 14 roles that mattered. The original found 10.

Source

I fine‑tuned a local AI model to judge job descriptions the way Claude does, then tested it on 116 roles it had never seen.

Source

The first run took about six hours on one desktop box.

Source

Version 1 is the one that fooled me, briefly. Its rank correlation tied the original. On a separate test set, the original's own correlation moved by 0.04 between two identical runs, so a gap of 0.002 is a tie.

Source

The fine‑tuned model never gave any of the 116 roles a score above 4.3. Not once.

Source