Treat Your References Like Function Arguments: A Developer's Mental Model for Video-to-Video AI

Description and use

In video‑to‑video generation the source clip becomes the contract, references are arguments, and prompts wire them. This function‑like approach reduces retries and improves predictability.

Problem

Video‑to‑video generation feels unstructured because the source video is treated as a suggestion rather than a specification, leading to noisy prompts and many retries.

Outcome

Using the function‑call mental model reduces prompt noise, improves consistency, and lowers the number of costly retries.

Workflow

• 1. Choose a source video that is a single continuous shot and contains the desired motion and framing.
• 2. Upload reference images, each dedicated to a single attribute (e.g., face, outfit).
• 3. Use positional reference syntax (e.g., @Image1, @Image2) in the prompt to map references to the source.
• 4. Select mode: "motion" to transfer movement or "swap" to replace a specific element.
• 5. Run a low‑resolution draft (480p) to validate identity, framing, and reference placement.
• 6. If satisfied, re‑render at higher resolution (720p/1080p).
• 7. Log each run in a JSON file with source, refs, prompt, mode, resolution, and notes for reproducibility.

Evidence

The key insight: the source video is not a suggestion, it is the bulk of your specification.

Source

Keep the prompt about mapping, not motion. Don’t redescribe the dance.

Source

Draft at low resolution. Run 480p to check identity, framing and that the right reference landed in the right place.

Source

If you want to poke at this without wiring up any infrastructure, I've been experimenting with Genjutsu, a browser-based video-to-video generator

Source