How I built Live Voice: one tool contract, six realtime voice providers

Description and use

Live Voice unifies six real‑time voice providers under a single tool contract, delivering a consistent UI and secure Cloudflare Worker relay.

Problem

Integrating multiple real‑time voice AI providers into a single mobile app while keeping a consistent tool‑based workflow and UI behavior.

Outcome

Live Voice now supports six real‑time providers with a single tool contract, consistent UI behavior, and a secure hosted relay, enabling users to log nutrition and exercise via voice across iOS and Android.

Workflow

• Define a single JSON contract listing 25 tool actions with declared effects (read, navigate, write, destructive).
• Implement adapters for each provider that translate between provider‑specific message formats and the contract.
• Ensure both iOS and Android apps bundle the same contract and CI verifies consistency.
• Add UI logic that interprets tool effects: show Undo for writes, require confirmation for destructive actions, and enforce user confirmation for writes when the setting is on.
• Deploy a Cloudflare Worker that acts as a WebSocket relay for the Plus/Pro plan, performing user‑id checks, rate limiting, and billing by time.
• Provide a Metal/AGSL orb UI element that visualizes listening/thinking/speaking states based on audio loudness.
• Publish the app and source code, inviting community feedback and PRs.

Evidence

The assistant can only act through tools. They live in one file, shared/voice‑assistant/tools.json: 25 actions like open_camera, capture_photo, scan_barcode, log_food, log_water, start_fast, log_workout, update_setting and end_conversation.

Source

Both apps bundle an identical copy, and a CI script fails the build if either copy drifts.

Source

The hosted relay… checks the user id, rate limits, the RevenueCat plan and the remaining voice balance.

Source

The glowing orb is a Metal shader on iOS and an AGSL port on Android 13+, with a Canvas fallback on older versions.

Source