I built an epistemic gate to stop LLM data poisoning during fine-tuning

Description and use

Beatriz is a lightweight proxy that defends against targeted data poisoning during LLM fine‑tuning. It verifies generated text against an immutable corpus using cosine similarity and a composite loss, achieving 93 % precision and 80 % recall on a multi‑domain benchmark.

Problem

Malicious data poisoning during fine‑tuning can silently corrupt specific facts while aggregate metrics improve, posing a stealthy threat to small teams and startups fine‑tuning local models.

Outcome

Across five architectures (GPT‑2, Qwen‑2.5‑0.5B, TinyLlama‑1.1B, Pythia‑1.4B, Phi‑3‑mini‑4k‑instruct) the gate achieved a robust truth margin (+4.19 ± 0.08) with 0.93 precision and 0.80 recall on a 30‑fact benchmark. Ablation showed that filtering alone provides 65 % of the benefit, with contrastive loss adding the remaining 35 %.

Workflow

• 1. Generate candidate text from a generative source. 2. Pass the text through the Beatriz Gate. 3. Compute cosine similarity against an immutable anchor corpus. 4. Apply a composite loss (cross‑entropy + contrastive Softplus) to enforce truthfulness. 5. Forward the filtered text to the student model for fine‑tuning.
• 6. Evaluate the fine‑tuned model on a held‑out multi‑domain benchmark to confirm robustness.

Evidence

Beatriz acts as a lightweight, non‑invasive defensive proxy (~0.1 ms decision latency) that sits between a generative source and a training student model.

Source

Key Results (EXP08–EXP16): Multi‑architecture validation: Tested across GPT‑2 (124M), Qwen‑2.5‑0.5B, TinyLlama‑1.1B, Pythia‑1.4B, and Phi‑3‑mini‑4k‑instruct (3.8B).

Source

Scaled Held‑Out (EXP16): Validated on a rigorous 30 multi‑domain unattacked facts benchmark, consolidating a robust truth margin of +4.19 ± 0.08 with 0.93 Precision and 0.80 Recall on the gate.

Source