Automated QA for 56,000 browser games: how we catch broken embeds before players do

Description and use

An automated nightly Playwright pipeline checks 56,000 browser games for broken embeds, reducing false positives to a few per week.

Problem

Maintaining a large catalog of embedded HTML5 games is impossible with manual QA; broken embeds cause player loss.

Outcome

The pipeline runs on a single modest server, covering the full catalog in ~6 hours nightly with concurrency 8, costing far less than a part‑time QA person. False positives dropped to a handful per week after rule refinements.

Workflow

• 1. Load game page and wait for embed iframe.
• 2. Verify iframe source returns HTTP 200; flag 404/403 immediately.
• 3. Wait for first paint inside iframe to confirm canvas rendering.
• 4. Listen for console errors; flag if many uncaught exceptions.
• 5. Capture screenshot of game area and diff against last known‑good image; flag if significant change.
• 6. If flagged, retain screenshots, logs, and last good date in review queue.
• 7. Human reviews queue each morning, deciding to fix embed URL, replace archived file, or delist game.

Evidence

Manual QA at this scale is impossible. Nobody is clicking through 56,000 games to check they still load.

Source

The screenshot diff was the breakthrough. HTTP checks alone missed about 30 percent of real failures, games that loaded fine technically but rendered nothing.

Source

This brought false positives down to a handful per week, each one worth a human look.

Source