Video Depth Estimation in the Browser: Building with WebGPU and Depth Anything V2

Description and use

Depth estimation is implemented in the browser using Depth‑Anything‑V2 and WebGPU, enabling depth‑aware effects, depth video export, and use as references for AI video generation, with noted limitations in motion accuracy and semantics.

Problem

Video editors lack explicit depth information in ordinary video, limiting depth‑aware effects and AI generation.

Outcome

Local depth estimation enables depth‑aware blur, relighting, and export of depth videos, and demonstrates that depth references can guide AI video generation, though with limitations in temporal stability and semantic fidelity.

Workflow

• Map timeline time to source time using getVisualSourceTime
• Decode and resize a frame
• Run depth inference in a Web Worker with Transformers.js (WebGPU, q4f16)
• Normalize and refine depth pixels
• Preview effect or encode depth video
• Export depth video as WebM VP9 (max 1080p)

Evidence

I added browser‑local depth estimation to Timeline Studio, my open‑source video editor, to turn that missing information into an editable resource.

Source

The worker loads the model through Transformers.js with WebGPU and a Q4F16 configuration.

Source

The analysis uses the editor's shared source‑time mapping: const sourceTime = getVisualSourceTime(segment, localTime);

Source

The current browser path uses WebM with VP9 video, preserves the source aspect ratio, and caps the longest output dimension at 1080 pixels.

Source

Some broad movements survived, including turning toward the camera and moving into a closer shot. But the results were not frame‑exact.

Source