Why we open-sourced cachegate, the LLM router we built for our own product

Description and use

Cachegate is an open-source LLM proxy that caches exact and semantic duplicate requests and routes to the cheapest and fastest providers, cutting costs by 20‑90%.

Problem

High LLM API costs due to repeated or near-duplicate requests.

Outcome

Cachegate reduced LLM spend by 20‑45% through caching and up to 90% when combined with routing, providing transparent, open-source tooling.

Workflow

• 1. Deploy cachegate as a proxy in front of LLM providers.
• 2. Point client baseURL to cachegate endpoint.
• 3. Configure capability tiers and routing strategy (cheapest, fastest, or latency-guardrail).
• 4. Cache exact matches and semantic near-duplicates using cosine similarity over capped Redis list.
• 5. Monitor cache hit rates via /stats endpoint.
• 6. Adjust strategy based on real spend savings (20‑45% from caching, 47‑90% with routing).

Evidence

Caching alone typically saves 20-45% of spend; adding routing on top of well-tuned traffic can reach 47-90%.

Source

Two kinds of cache, reported honestly. An exact-match cache and a semantic cache for near-duplicates.

Source