Fine-tuning Pocket TTS for Castilian Spanish: stop at step 600 and judge the accent by ear

Description and use

Fine‑tuning Pocket TTS for Castilian Spanish stopped at step 600 using EMA weights, preserving intelligibility and correct accent while avoiding word loss and errors.

Problem

The base Spanish Pocket TTS model sounded like Latin America, not Castilian Spain, and training longer degraded quality.

Outcome

The fine‑tuned model at step 600 with EMA weights delivers Castilian Spanish narration on CPU, matching base intelligibility and avoiding word errors, while keeping generation speed acceptable.

Workflow

• 1. Choose 24‑layer Spanish model for speed trade‑off.
• 2. Fine‑tune on ~2,000 Castilian recordings using AMD GPU ROCm.
• 3. Monitor validation loss; stop at step 600 where loss 0.4951.
• 4. Release EMA weights instead of raw checkpoint.
• 5. Deploy with pocket-tts serve --config hf://inferencekey/pocket-tts-spanish-24l-castilian/config.yaml.
• 6. Generate audio per complete sentence, spell numbers out.
• 7. Evaluate with Qwen3‑ASR to catch dropped/invented words.
• 8. Have a human listen to each clip to confirm accent.
• 9. Iterate if ASR fails or accent unsatisfactory.

Evidence

The base Spanish model was perfectly intelligible. The accent gave it away: foreign‑sounding, with seseo, where the Castilian "z" and soft "c" collapse into an "s" as in much of Latin America.

Source

Validation loss is the error the model makes on held‑out clips it never trains on; lower is better. Ours hit its minimum at step 600 (0.4951) and had drifted up to 0.5095 by step 1500.

Source

From step 600 we released the EMA weights instead of the raw ones. EMA (exponential moving average) blends the weights over recent steps, so it moves further from the base model more gently.

Source

First, send complete sentences or short paragraphs. The model ends each chunk with a falling, final intonation, so a fragmented paragraph sounds stop‑and‑go.

Source

Second, spell numbers out. The model handles "siete y cuarto" every time; "7:15" sometimes trips it.

Source