I built a hands-free slide remote for a friend who can never find the right page

Description and use

Presenters often lose track of slide position in large PDFs. A local remote was built using MediaPipe hand tracking, Whisper speech‑to‑text, and Gemma for semantic slide selection, operating entirely offline for privacy and cost savings.

Problem

Presenters lose track of slide position in large PDFs during talks, making navigation stressful and error-prone.

Outcome

The remote reliably navigates slides using gestures and voice, with a hybrid architecture that reduces reliance on LLMs for simple commands and improves safety by preferring no‑action over incorrect jumps.

Workflow

• Extract slide text from PDF/PPTX
• Index slides
• Track hand via MediaPipe from webcam
• Detect gestures (swipe, open palm, fist) and map to slide actions
• Record audio via microphone
• Transcribe with Whisper using slide titles as hints
• Detect command phrases
• Apply deterministic logic for simple commands
• Apply keyword + fuzzy matching for content requests
• If multiple candidates, send to Gemma via Ollama for final decision
• Execute slide navigation or no‑action if confidence low

Evidence

The webcam tracks the presenter's hand using MediaPipe.

Source

Whisper runs locally, so the microphone audio does not need to be uploaded to a speech API.

Source

Gemma via Ollama → semantic understanding and slide selection

Source