Simon Sees

说明与用途

Simon Sees 是一款户外 Simon Says 的实时裁判系统，使用 SAM 3、DINOv3 和 Qwen3.5‑4B 模型，让玩家无需关注屏幕即可持续游戏。

问题

Playground games end when people check phones; no live referee for group play.

结果

In replay demos, the fine‑tuned referee achieved 87.4 % accuracy on a hand‑checked test set, and in a recorded clip the middle player correctly performed "hands on head" with probabilities 0.00, 0.995, 0.00, leading to a win.

工作流

• Prop a phone on a stump or backpack, pointed at players
• Line up players and tap "Line up" to generate photo cards
• Phone streams frames to laptop
• SAM 3 segments each person in each frame
• DINOv3 embeds each crop and matches to line‑up gallery via Hungarian assignment
• Qwen3.5‑4B fine‑tuned on Tinker judges each crop for the posed question
• Referee returns probability; game applies rules and announces eliminations
• Simon calls out commands and results
• Last player standing wins

证据

On October 8 there were 321 valid Week 1 entries, and 130 of them were outdoor quests, missions, scavenger hunts or games.

来源

The phone streams a few frames a second to a laptop. SAM 3 finds every person, DINO embeddings work out which player is which by comparing each one with their line‑up photo, and a referee looks at each player's own crop and answers one yes/no question.

来源

The fine‑tuned referee gave the three players P = 0.00, 0.995 and 0.00. Only the middle player had her hands on her head, so she wins after one round.

来源

The data. COCO 2017 person keypoints gave me labels without hand‑labelling thousands of photos.

来源