FitGuru: Building a Private, On-Device AI Workout Coach in Kotlin

Описание и применение

FitGuru демонстрирует, как смартфон может анализировать форму упражнений и считать повторения, не отправляя видео в облако, используя MediaPipe Pose для локального распознавания поз и Gemini для пост‑анализов.

Задача

How to coach workout form and count reps on a phone without sending video to the cloud?

Результат

FitGuru demonstrates that a phone can coach squats and other exercises, count reps accurately, provide real‑time corrections, and give a post‑set debrief—all while keeping video on the device.

Рабочий процесс

• Open app and select lift
• Place phone to capture user
• CameraX streams frames to MediaPipe Pose
• Reflex Engine computes joint angles and runs state machines for rep counting and form cues
• Spoken corrections are delivered via TTS or ElevenLabs voice during the set
• User says "finish set" to trigger Cognitive Engine
• Cognitive Engine (Gemini 2.5 Flash) processes summary JSON and generates a spoken debrief
• History is stored in Room database
• Optional ElevenLabs audio is used if API key provided

Доказательства

We built a phone-side workout coach at Innovhacks 4.0 - MediaPipe pose, joint-angle state machines, Gemini debriefs, and ElevenLabs voice - without sending your camera feed to the cloud.

Источник

If the camera feed leaves the phone, we already lost.

Источник

A squat is not a guess. A rep exists only when the knee and hip cross a “down” threshold and then return past an “up” threshold.

Источник