Simon Sees

Описание и применение

Simon Sees — это система рефери для игры «Simon Says» на открытой площадке, использующая модели SAM 3, DINOv3 и Qwen3.5‑4B, позволяющая игрокам продолжать игру без отвлечения на экран.

Задача

Playground games end when people check phones; no live referee for group play.

Результат

In replay demos, the fine‑tuned referee achieved 87.4 % accuracy on a hand‑checked test set, and in a recorded clip the middle player correctly performed "hands on head" with probabilities 0.00, 0.995, 0.00, leading to a win.

Рабочий процесс

• Prop a phone on a stump or backpack, pointed at players
• Line up players and tap "Line up" to generate photo cards
• Phone streams frames to laptop
• SAM 3 segments each person in each frame
• DINOv3 embeds each crop and matches to line‑up gallery via Hungarian assignment
• Qwen3.5‑4B fine‑tuned on Tinker judges each crop for the posed question
• Referee returns probability; game applies rules and announces eliminations
• Simon calls out commands and results
• Last player standing wins

Доказательства

On October 8 there were 321 valid Week 1 entries, and 130 of them were outdoor quests, missions, scavenger hunts or games.

Источник

The phone streams a few frames a second to a laptop. SAM 3 finds every person, DINO embeddings work out which player is which by comparing each one with their line‑up photo, and a referee looks at each player's own crop and answers one yes/no question.

Источник

The fine‑tuned referee gave the three players P = 0.00, 0.995 and 0.00. Only the middle player had her hands on her head, so she wins after one round.

Источник

The data. COCO 2017 person keypoints gave me labels without hand‑labelling thousands of photos.

Источник