FitGuru: Building a Private, On-Device AI Workout Coach in Kotlin

Description and use

FitGuru shows how a phone can coach workouts, count reps, and provide feedback without sending video to the cloud, using on‑device MediaPipe Pose and Gemini for post‑set analysis.

Problem

How to coach workout form and count reps on a phone without sending video to the cloud?

Outcome

FitGuru demonstrates that a phone can coach squats and other exercises, count reps accurately, provide real‑time corrections, and give a post‑set debrief—all while keeping video on the device.

Workflow

• Open app and select lift
• Place phone to capture user
• CameraX streams frames to MediaPipe Pose
• Reflex Engine computes joint angles and runs state machines for rep counting and form cues
• Spoken corrections are delivered via TTS or ElevenLabs voice during the set
• User says "finish set" to trigger Cognitive Engine
• Cognitive Engine (Gemini 2.5 Flash) processes summary JSON and generates a spoken debrief
• History is stored in Room database
• Optional ElevenLabs audio is used if API key provided

Evidence

We built a phone-side workout coach at Innovhacks 4.0 - MediaPipe pose, joint-angle state machines, Gemini debriefs, and ElevenLabs voice - without sending your camera feed to the cloud.

Source

If the camera feed leaves the phone, we already lost.

Source

A squat is not a guess. A rep exists only when the knee and hip cross a “down” threshold and then return past an “up” threshold.

Source