Fine-tuning a 0.8B model to beat hosted LLMs at one job

Description and use

Fine‑tuning a 0.8B model achieved ROC AUC 0.954 on memory‑update decisions, outperforming hosted LLMs.

Problem

When a new fact about someone arrives, the system must decide whether to replace an existing fact, retain it, or ignore it, to avoid stale or over‑eager updates in long‑term memory.

Outcome

The fine‑tuned 0.8B model achieved ROC AUC 0.954 on stated updates, surpassing hosted LLMs such as Gemini 3.7 Flash (0.892) and Gemma 4 31B (0.887). At a 0.70 threshold it never replaced a still‑true fact (precision 1.00) but had low recall (0.14). Training required 12 GPU‑hours on a single T4.

Workflow

• Collect 6,818 logged Jev decisions and 10,219 simulated decisions with ground truth; build 6,000 training examples prioritizing real updates.
• Stage 1: supervised fine‑tuning with LoRA on Qwen3.5‑0.8B‑Base, rank 16, alpha 32, dropout 0.05, learning rate 5e‑5, one epoch over 6,000 examples on a single NVIDIA T4.
• Stage 2: reward‑ranked RL (ReST) using stage‑1 model decisions on 2,000 training decisions, scoring pairs with a custom reward, relabeling 52 negative cases, and fine‑tuning with learning rate 1e‑5.
• Validate on six separate lives (400 decisions) and select checkpoint with highest AUC.
• Test on two held‑out sets: a two‑year simulated life and a live run with LLM‑generated facts, measuring ROC AUC, precision, and recall at 0.70 threshold.
• Deploy the model via Hugging Face or Kaggle, compatible with any Jev client.

Evidence

The short version: a 0.8B open model, fine‑tuned on 6,000 examples on a single T4, beat every hosted LLM I tested at this one job.

Source

At 0.70 or above, the old fact is replaced.

Source

The GPU was a single NVIDIA T4 with 16 GB. It has no bf16, so everything ran in fp32, with batch 1, gradient accumulation of 8 and gradient checkpointing.

Source

On the live test, stated updates went from 0.752 as released to 0.929 after stage 1 and 0.945 after stage 2.

Source

Averaged over both tests on stated updates, with every hosted LLM given the same definition: my fine‑tuned Kev (kev‑mem): 0.954

Source