How our Top 11 HopHacks project turned noisy MediaPipe pose data into reliable rehabilitation metrics for adaptive AI planning

说明与用途

Mendly 将 MediaPipe Pose 的噪声数据转化为可靠的康复指标，处理无效观测、校正相机纵横比、使用状态机计数重复，并区分有效与低置信度数据。

问题

Rehabilitation exercises need reliable movement data, but MediaPipe pose estimation is noisy and unreliable, leading to inaccurate joint angles, repetitions, and performance metrics.

结果

By treating failed inferences as invalid, correcting for camera aspect ratio, adopting state‑based repetition detection, and separating valid from low‑confidence data, the team produced reliable rehabilitation metrics that improved adaptive AI planning.

工作流

• Capture camera frames
• Run MediaPipe Pose to get normalized landmarks
• Convert landmarks to joint angles using dot product
• Apply camera aspect‑ratio correction before geometry calculations
• Check landmark validity; if pose fails, mark observation invalid and skip angle calculation
• Run reliability checks on angles; if invalid, skip repetition update
• Use state machine to detect full repetitions (start, reach target, hold, reverse, return)
• Count only valid repetitions toward exercise target
• Record low‑confidence attempts separately for history but exclude from ROM statistics
• Monitor camera setup with rolling window; revoke Ready state if unreliable; timeout after ~25s to prompt retry

证据

We changed the pipeline to represent failed inference as an invalid observation.

来源

We fixed this by accounting for the actual camera aspect ratio before calculating angles and ray lengths.

来源

A rep has a beginning, progression, target, and completion.

来源

Automatic completion now depends on the number of valid repetitions, not the total number of attempts.

来源

We built a setup checker to decide whether the current view was usable for the exercise.

来源