Offline On-Device AI Agents with Gemma & Local Tool Calls (feat. Shivay Lamba)

Description and use

Doctors use local STT and Gemma models to transcribe and generate prescriptions offline, avoiding cloud data transfer and reducing latency.

Problem

Privacy and latency concerns when doctors send patient audio and medical records to cloud LLMs for prescription generation.

Outcome

Prescription order is generated and queued locally, eliminating cloud transmission and reducing wait times.

Workflow

• 1. Doctor records consultation and prescription notes.
• 2. On-device Speech-to-Text transcribes audio to text offline.
• 3. Gemma (12B) parses transcript, extracts medication order, and calls internal pharmacy API locally.
• 4. Pharmacy API queues prescription, ready for patient pickup.

Evidence

When a doctor consults with a patient and writes a prescription, traditional cloud LLM pipelines require sending sensitive patient audio and medical records over the network, or handing the patient a physical slip to wait in line at a pharmacy.

Source

Instead of routing sensitive patient conversations to external cloud APIs, Shivay demonstrates a two-stage local workflow running directly on a doctor's laptop or smartphone:

Source

1. Consultation Transcription – Local Speech-to-Text (STT) – On-Device (Laptop / Phone) – Transcribes the doctor-patient consultation and prescription notes completely offline.

Source

1. Structured Tool Calling – Gemma (12B Parameter Model) – On-Device Local Runtime – Parses the completed transcript, extracts the medication order, and triggers an automated pharmacy tool call.

Source

1. Pharmacy Fulfillment – Internal Order API – Local / Hospital Network – Queues the prescription so the order is ready for pickup by order number as soon as the patient arrives.

Source