Reading a mobile game screen pixel by pixel

Description and use

A dataset of 4.2M masks for Clash of Clans screens enabled training a segmentation model that accurately detects the no‑deploy zone with 0.996 IoU.

Problem

Clash of Clans players need to identify loot, troop placement, and deploy zones within a 30‑second window, but existing datasets lack pixel‑level annotations for the game’s complex UI and overlapping objects.

Outcome

The deploy zone model achieved 0.996 IoU on the validation set, accurately delineating the no‑deploy area and preventing mask leakage onto the army bar.

Workflow

• 1. Capture screenshots at three stages: scout screen, during battle, end of battle.
• 2. Annotate each visible object as a polygon mask, accounting for overlapping and partial visibility.
• 3. Compile 6,909 frames into 4.2M masks, split validation set by base to avoid memorization.
• 4. Train a U‑Net segmentation model (MobileNetV2 or EfficientNet‑B4) to predict the no‑deploy area mask.
• 5. Evaluate on validation set, achieving 0.996 IoU for the no‑deploy area.

Evidence

I built one.

Source

6,909 frames, around 4.2M masks. YOLO seg and COCO.

Source

0.996 IoU on the no‑deploy area on val.

Source