Real-time Local ASL Interpreter with Gemma 4 and MediaPipe

Description and use

The system achieved an overall accuracy of 59.1% on a test set of 66 images. Letters B, D, E, F, G, H, K, R, T, U, X, Y, Z achieved 100% accuracy. Letters I, O, and P had 0% accuracy, while V had 22%. The model correctly flagged motion-based signs (J, Z) as low confidence when presented as still images. Live testing showed 3 correct and 12 incorrect predictions in 15 sessions, with common errors involving O being predicted instead of A, C, or E.

Problem

Most American Sign Language (ASL) recognition tools require cloud APIs, compromising privacy, or rely on dedicated pre-trained models that limit flexibility. The goal was to build a privacy-preserving, local ASL interpreter using a general-purpose vision model without fine-tuning.

Outcome

The system achieved an overall accuracy of 59.1% on a test set of 66 images. Letters B, D, E, F, G, H, K, R, T, U, X, Y, Z achieved 100% accuracy. Letters I, O, and P had 0% accuracy, while V had 22%. The model correctly flagged motion-based signs (J, Z) as low confidence when presented as still images. Live testing showed 3 correct and 12 incorrect predictions in 15 sessions, with common errors involving O being predicted instead of A, C, or E.

Workflow

• Capture video feed from the local webcam.
• Use MediaPipe in the browser to detect the hand and crop a tight 512x512 square around it, removing background noise.
• Send the cropped hand image to the local Gemma 4 (e4b) model.
• Use a prompt containing specific descriptions of all 26 ASL letters and explicit disambiguation rules for commonly confused pairs (e.g., A vs S, M vs N vs T).
• Gemma 4 analyzes the image and returns the predicted letter, confidence score, and observations about hand position.
• Log results to a timestamped CSV for evaluation, including user feedback (thumbs up/down) for incorrect predictions.

Evidence

Overall: 59.1% ... Best letters: B, D, E, F, G, H, K, R, T, U, X, Y, Z -> 100% ... Worst letters: I (0%), O (0%), P (0%), V (22%)

Source

The fix was adding MediaPipe as a pre-processing step... Separating those two jobs made the biggest accuracy difference of anything I tried.

Source