Local-First LLM Routing: A Decision Table for Latency, Secrets, and Offline Mode

Описание и применение

Руководство по внедрению локального маршрутизатора LLM, который решает, где обрабатывать запросы, учитывая задержку, конфиденциальность и офлайн‑режим.

Задача

Field‑service chatbots that always send requests to a cloud LLM can suffer long backlogs, expose PII in third‑party logs, and fail when offline.

Результат

The router reduced latency backlogs, prevented PII leakage, and ensured availability offline. Shadow‑run logs revealed that 80% of requests could run locally, turning the cloud leg into a fallback.

Рабочий процесс

• 1. Classify payload as public, internal, or PII before routing.
• 2. Probe network connectivity with a hard timeout.
• 3. Evaluate request against the decision table (latency, complexity, queue depth, token budget).
• 4. Route to local, cloud, or queue based on the decision.
• 5. On local failure, retry on cloud if payload is not PII; on cloud failure, fallback to local.
• 6. Run a shadow week in dry‑run mode, logging decisions to a JSONL file.
• 7. Aggregate logs to confirm expected split and compare p95 latencies.
• 8. Cut over to the new router after logs validate the split.
• 9. Use the JSONL log as regression fixture for future policy changes.
• 10. Run unit tests with pytest to lock key routing rules.

Доказательства

A field‑service team built a support chatbot that sent every message to a cloud LLM endpoint. The design held until a technician drove through a tunnel, and the request queue grew into an eleven‑minute backlog.

Источник

The same week, a support ticket containing a customer's account number appeared in a third‑party log because the payload was never classified.

Источник

The router implementation in the next section turns that table into executable logic with a small Python module.

Источник