Your auto-captioner is measuring the wrong thing: a ratio that fired on a perfectly good clip

Description and use

Automatic subtitle generation for AI videos: fixing sync and overly long captions by aligning script text with Whisper timing and filtering segments with low speech rate.

Problem

Automatic captioning of AI‑generated ad videos produced out‑of‑sync or overly long subtitles due to mismatched timing and text sources.

Outcome

Accurate subtitles that match the spoken content and timing, eliminating long static captions and mis‑aligned text.

Workflow

• Generate video clip
• Extract audio and run Whisper transcription to get timing and text
• Align script lines with Whisper segments using word‑count weighting
• Generate .ass file with aligned timings and script text
• Burn subtitles with ffmpeg
• Apply speech‑rate guard: if any segment’s words‑per‑second < 1.5, fallback to hand‑written cues

Evidence

Take timing from the audio, take text from the script.

Source

If any single segment covers more than 40% of the clip, distrust the timing

Source