Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

Description and use

Fine‑tuning Qwen2.5‑1.5B on a Mac eliminated repetitive chatbot responses, reduced prompt size, and improved speed while maintaining answer quality.

Problem

The chatbot repeatedly parrots the same answer to different user prompts, ignoring long system rules and producing off‑topic or invented responses.

Outcome

The fine‑tuned model reduces repeated and invented answers, improves prompt processing time by ~50%, and maintains or improves response quality while keeping the system lightweight for CPU‑only deployment.

Workflow

• 1. Generate training data with a local teacher (Qwen2.5‑14B‑Instruct on M4 Pro) using seed questions and paraphrases, filtering junk and saving 828 records.
• 2. Review records by hand with a CLI linter, accepting 686 and rejecting 142 for invented claims or instruction leaks.
• 3. Fine‑tune with LoRA using MLX: 16 top layers, 4‑batch, 1e‑4 LR, 463 iterations, 31 minutes, 10.55M trainable params.
• 4. Fuse adapter into base model and convert to GGUF (f16 then Q4_K_M).
• 5. Evaluate against baseline on 5 held‑out conversations: flagged answers reduced from 9/7 to 1/1 (temp 0.3) and 0/0 (temp 0.6).
• 6. Publish model to Hugging Face and update CDK to point to new GGUF URL, triggering EC2 redeploy.

Evidence

Three different messages, "Tell me a joke", "could you tell me a joke?" and "can you help with ats?", got the same answer word for word.

Source

Prompt size went from 1,444 tokens (system prompt plus reminder) to 788.

Source

Five held‑out conversations (21 answers), never used in training, including the original joke/ATS conversation.

Source