Why I chose on-device inference for a writing assistant

说明与用途

DraftKey 选择本地推理，以确保用户文本不离开 Mac、离线使用且无 API 成本，但需要 1.1 GB 模型、8 GB RAM 和 Apple Silicon 机器。

问题

Deciding between local model inference and cloud API for a writing assistant to balance privacy, cost, and performance.

结果

On-device inference provided strong privacy, offline capability, and no API costs, with acceptable performance for short rewrites and autocomplete, though it required significant local resources.

工作流

• 1. Evaluate privacy and cost implications of cloud API.
• 2. Assess hardware requirements for local inference (1.1 GB model, 8 GB RAM, Apple Silicon).
• 3. Choose a small local model sufficient for short rewrites and autocomplete.
• 4. Deploy model locally within the app.
• 5. Test performance and user experience.
• 6. Release DraftKey with on-device inference.
• 7. Gather user feedback on privacy and functionality.

证据

The upside: people's text never leaves their Mac, it works offline, and there are no per-request API costs.

来源

The downside: the app needs a model download of about 1.1 GB, at least 8 GB of RAM, and an Apple Silicon Mac, and a small model has less raw capability than the frontier ones.

来源

For short rewrites and autocomplete, a small local model turned out to be good enough, and privacy ended up being the feature people notice most.

来源