How our Top 11 HopHacks project turned noisy MediaPipe pose data into reliable rehabilitation metrics for adaptive AI planning

Описание и применение

Проект Mendly преобразовал шумные данные MediaPipe Pose в надёжные метрики реабилитации, исправив обработку недостоверных наблюдений, учитывая аспект‑соотношение камеры, используя состояние для подсчёта повторений и отделяя валидные попытки от низкокачественных.

Задача

Rehabilitation exercises need reliable movement data, but MediaPipe pose estimation is noisy and unreliable, leading to inaccurate joint angles, repetitions, and performance metrics.

Результат

By treating failed inferences as invalid, correcting for camera aspect ratio, adopting state‑based repetition detection, and separating valid from low‑confidence data, the team produced reliable rehabilitation metrics that improved adaptive AI planning.

Рабочий процесс

• Capture camera frames
• Run MediaPipe Pose to get normalized landmarks
• Convert landmarks to joint angles using dot product
• Apply camera aspect‑ratio correction before geometry calculations
• Check landmark validity; if pose fails, mark observation invalid and skip angle calculation
• Run reliability checks on angles; if invalid, skip repetition update
• Use state machine to detect full repetitions (start, reach target, hold, reverse, return)
• Count only valid repetitions toward exercise target
• Record low‑confidence attempts separately for history but exclude from ROM statistics
• Monitor camera setup with rolling window; revoke Ready state if unreliable; timeout after ~25s to prompt retry

Доказательства

We changed the pipeline to represent failed inference as an invalid observation.

Источник

We fixed this by accounting for the actual camera aspect ratio before calculating angles and ray lengths.

Источник

A rep has a beginning, progression, target, and completion.

Источник

Automatic completion now depends on the number of valid repetitions, not the total number of attempts.

Источник

We built a setup checker to decide whether the current view was usable for the exercise.

Источник