The fine‑tune that looked fine

Описание и применение

Фин‑тюнинг локальной модели не смог воспроизвести точность Claude в оценке вакансий: хотя корреляция рангов была сопоставима, модель пропустила 8 из 14 сильных ролей и не улучшила число ложных тревог. Оригинальная модель осталась в продакшене.

Задача

Can a fine‑tuned local AI model replace an expensive Claude model for job‑description triage while matching its decision quality?

Результат

Fine‑tuning reduced prompt length but failed to match Claude on strong‑role recall; the original model remains in production.

Рабочий процесс

• Collect 1,160 Claude evaluations (job description, score, reasoning).
• Hold out 116 evaluations (14 strong) for testing.
• Train three fine‑tuned models on the remaining 1,044: V1 repeats strong roles 3×, V2 repeats strong roles 6× and near‑strong 2×, V3 trains on scores only.
• Evaluate each model on the held‑out set, measuring rank correlation, strong‑role recall, false alarms, and big misses.
• Analyze score distributions and attempt post‑processing adjustments (shift, stretch, linear fit, cutoff change).
• Conclude that fine‑tuning did not improve strong‑role recall and the original model remains in production.

Доказательства

My fine‑tuned AI model matched the original on the headline metric and cut false alarms by more than half. It also found 6 of the 14 roles that mattered. The original found 10.

Источник

I fine‑tuned a local AI model to judge job descriptions the way Claude does, then tested it on 116 roles it had never seen.

Источник

The first run took about six hours on one desktop box.

Источник

Version 1 is the one that fooled me, briefly. Its rank correlation tied the original. On a separate test set, the original's own correlation moved by 0.04 between two identical runs, so a gap of 0.002 is a tie.

Источник

The fine‑tuned model never gave any of the 116 roles a score above 4.3. Not once.

Источник