Why I chose on-device inference for a writing assistant

Description and use

DraftKey chose on-device inference to keep user text private, enable offline use, and avoid API costs, at the expense of a large model download, 8 GB RAM, and Apple Silicon requirement.

Problem

Deciding between local model inference and cloud API for a writing assistant to balance privacy, cost, and performance.

Outcome

On-device inference provided strong privacy, offline capability, and no API costs, with acceptable performance for short rewrites and autocomplete, though it required significant local resources.

Workflow

• 1. Evaluate privacy and cost implications of cloud API.
• 2. Assess hardware requirements for local inference (1.1 GB model, 8 GB RAM, Apple Silicon).
• 3. Choose a small local model sufficient for short rewrites and autocomplete.
• 4. Deploy model locally within the app.
• 5. Test performance and user experience.
• 6. Release DraftKey with on-device inference.
• 7. Gather user feedback on privacy and functionality.

Evidence

The upside: people's text never leaves their Mac, it works offline, and there are no per-request API costs.

Source

The downside: the app needs a model download of about 1.1 GB, at least 8 GB of RAM, and an Apple Silicon Mac, and a small model has less raw capability than the frontier ones.

Source

For short rewrites and autocomplete, a small local model turned out to be good enough, and privacy ended up being the feature people notice most.

Source