How our Top 11 HopHacks project turned noisy MediaPipe pose data into reliable rehabilitation metrics for adaptive AI planning
Description and use
Mendly turned noisy MediaPipe pose data into reliable rehabilitation metrics by handling invalid observations, correcting for camera aspect ratio, using state‑based repetition detection, and separating valid from low‑confidence data.
Problem
Rehabilitation exercises need reliable movement data, but MediaPipe pose estimation is noisy and unreliable, leading to inaccurate joint angles, repetitions, and performance metrics.
Outcome
By treating failed inferences as invalid, correcting for camera aspect ratio, adopting state‑based repetition detection, and separating valid from low‑confidence data, the team produced reliable rehabilitation metrics that improved adaptive AI planning.
Workflow
- Capture camera frames
- Run MediaPipe Pose to get normalized landmarks
- Convert landmarks to joint angles using dot product
- Apply camera aspect‑ratio correction before geometry calculations
- Check landmark validity; if pose fails, mark observation invalid and skip angle calculation
- Run reliability checks on angles; if invalid, skip repetition update
- Use state machine to detect full repetitions (start, reach target, hold, reverse, return)
- Count only valid repetitions toward exercise target
- Record low‑confidence attempts separately for history but exclude from ROM statistics
- Monitor camera setup with rolling window; revoke Ready state if unreliable; timeout after ~25s to prompt retry
Evidence
We changed the pipeline to represent failed inference as an invalid observation.
We fixed this by accounting for the actual camera aspect ratio before calculating angles and ray lengths.
A rep has a beginning, progression, target, and completion.
Automatic completion now depends on the number of valid repetitions, not the total number of attempts.
We built a setup checker to decide whether the current view was usable for the exercise.
- Record date: 2026-10-01T22:55:56.790Z