Reading a mobile game screen pixel by pixel

Описание и применение

Создан датасет с 4,2 млн масок для экрана Clash of Clans, позволивший обучить сегментацию зоны, где нельзя размещать войска, с точностью 0,996 IoU.

Задача

Clash of Clans players need to identify loot, troop placement, and deploy zones within a 30‑second window, but existing datasets lack pixel‑level annotations for the game’s complex UI and overlapping objects.

Результат

The deploy zone model achieved 0.996 IoU on the validation set, accurately delineating the no‑deploy area and preventing mask leakage onto the army bar.

Рабочий процесс

• 1. Capture screenshots at three stages: scout screen, during battle, end of battle.
• 2. Annotate each visible object as a polygon mask, accounting for overlapping and partial visibility.
• 3. Compile 6,909 frames into 4.2M masks, split validation set by base to avoid memorization.
• 4. Train a U‑Net segmentation model (MobileNetV2 or EfficientNet‑B4) to predict the no‑deploy area mask.
• 5. Evaluate on validation set, achieving 0.996 IoU for the no‑deploy area.

Доказательства

I built one.

Источник

6,909 frames, around 4.2M masks. YOLO seg and COCO.

Источник

0.996 IoU on the no‑deploy area on val.

Источник