Why we open-sourced cachegate, the LLM router we built for our own product

说明与用途

Cachegate 是一个开源 LLM 代理，缓存精确和语义重复请求，并将请求路由到最便宜、最快的提供商，降低成本 20‑90%。

问题

High LLM API costs due to repeated or near-duplicate requests.

结果

Cachegate reduced LLM spend by 20‑45% through caching and up to 90% when combined with routing, providing transparent, open-source tooling.

工作流

• 1. Deploy cachegate as a proxy in front of LLM providers.
• 2. Point client baseURL to cachegate endpoint.
• 3. Configure capability tiers and routing strategy (cheapest, fastest, or latency-guardrail).
• 4. Cache exact matches and semantic near-duplicates using cosine similarity over capped Redis list.
• 5. Monitor cache hit rates via /stats endpoint.
• 6. Adjust strategy based on real spend savings (20‑45% from caching, 47‑90% with routing).

证据

Caching alone typically saves 20-45% of spend; adding routing on top of well-tuned traffic can reach 47-90%.

来源

Two kinds of cache, reported honestly. An exact-match cache and a semantic cache for near-duplicates.

来源