Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

Описание и применение

Финетюнинг модели Qwen2.5‑1.5B на Mac позволил устранить повторяющиеся ответы чат‑бота, сократить размер системного запроса и ускорить обработку, сохранив при этом качество ответов.

Задача

The chatbot repeatedly parrots the same answer to different user prompts, ignoring long system rules and producing off‑topic or invented responses.

Результат

The fine‑tuned model reduces repeated and invented answers, improves prompt processing time by ~50%, and maintains or improves response quality while keeping the system lightweight for CPU‑only deployment.

Рабочий процесс

• 1. Generate training data with a local teacher (Qwen2.5‑14B‑Instruct on M4 Pro) using seed questions and paraphrases, filtering junk and saving 828 records.
• 2. Review records by hand with a CLI linter, accepting 686 and rejecting 142 for invented claims or instruction leaks.
• 3. Fine‑tune with LoRA using MLX: 16 top layers, 4‑batch, 1e‑4 LR, 463 iterations, 31 minutes, 10.55M trainable params.
• 4. Fuse adapter into base model and convert to GGUF (f16 then Q4_K_M).
• 5. Evaluate against baseline on 5 held‑out conversations: flagged answers reduced from 9/7 to 1/1 (temp 0.3) and 0/0 (temp 0.6).
• 6. Publish model to Hugging Face and update CDK to point to new GGUF URL, triggering EC2 redeploy.

Доказательства

Three different messages, "Tell me a joke", "could you tell me a joke?" and "can you help with ats?", got the same answer word for word.

Источник

Prompt size went from 1,444 tokens (system prompt plus reminder) to 788.

Источник

Five held‑out conversations (21 answers), never used in training, including the original joke/ATS conversation.

Источник