Why we open-sourced cachegate, the LLM router we built for our own product

Описание и применение

Cachegate – открытый прокси для LLM, который кэширует точные и семантические дубликаты запросов и маршрутизирует к самым дешевым и быстрым провайдерам, снижая расходы на 20‑90%.

Задача

High LLM API costs due to repeated or near-duplicate requests.

Результат

Cachegate reduced LLM spend by 20‑45% through caching and up to 90% when combined with routing, providing transparent, open-source tooling.

Рабочий процесс

• 1. Deploy cachegate as a proxy in front of LLM providers.
• 2. Point client baseURL to cachegate endpoint.
• 3. Configure capability tiers and routing strategy (cheapest, fastest, or latency-guardrail).
• 4. Cache exact matches and semantic near-duplicates using cosine similarity over capped Redis list.
• 5. Monitor cache hit rates via /stats endpoint.
• 6. Adjust strategy based on real spend savings (20‑45% from caching, 47‑90% with routing).

Доказательства

Caching alone typically saves 20-45% of spend; adding routing on top of well-tuned traffic can reach 47-90%.

Источник

Two kinds of cache, reported honestly. An exact-match cache and a semantic cache for near-duplicates.

Источник