How I QA a Video Editor With One Test That Uses the App Like a Person

Description and use

A single Appium script simulates user actions, compares screenshots with SSIM, and validates export properties with ffprobe to ensure a video editor’s preview matches the final output.

Problem

Unit tests miss visual glitches and export mismatches in a video editor.

Outcome

The single end‑to‑end test reliably detects preview glitches, export mismatches, and ensures exported videos meet strict SDR and format requirements, preventing user‑reported bugs.

Workflow

• Run appium/run.sh to lease a simulator, pin UI elements, build the app, populate Photos library with ffmpeg‑generated clips, start Appium server, and launch pytest journey.
• Use Appium to perform real taps, drags, and sheet interactions, waiting for state changes via the hidden uitest‑state element.
• Take screenshots at key moments, ensuring two identical captures, exact frame alignment, and masking of animated tiles, then compare with approved baselines using SSIM thresholds.
• Pause editor at 1, 5.5, 8, and 11.5 s, capture canvas frames, export the project, extract frames from the .mp4, and compare each pair.
• Pull exported file from app container and validate codec, resolution, duration, audio track, and absence of black frames using ffprobe.

Evidence

Unit tests can't see a preview that blinks or an export that drifts from what you edited.

Source

It opens the app on a fresh simulator, picks clips in the Photos picker, trims, splits, adds a transition, generates captions, styles them, adds a text overlay, plays the project, exports it and opens the share screen.

Source

Under test, the editor mounts a hidden accessibility element called uitest-state.

Source

At key moments the journey takes a screenshot and compares it with an approved baseline using SSIM. The threshold is 0.985.

Source

During the journey, the test pauses the editor at 1, 5.5, 8 and 11.5 seconds and saves what the canvas shows. After the export, it pulls the frames at the same instants out of the .mp4 and compares each pair.

Source

The export is pulled out of the app's container and checked with ffprobe: H.264, SDR (bt709), 8‑bit 4:2:0, 1080×1920 at 30 fps, duration within one frame of the timeline, an audio track is present, no flat black frame anywhere in the video.

Source