Case Study: A C++ Cache Proxy That Cut Free-Tier Model Calls by 73%

Description and use

A custom C++ cache proxy reduced free-tier LLM API calls by 73% in a simulated CI workload, cutting median latency from 1.8s to 12ms for repeated requests and extending quota utility by 3.7x.

Problem

Free-tier model quotas are wasted on duplicate requests in CI pipelines, where identical compiler errors or code snippets trigger redundant model calls, consuming quota unnecessarily.

Outcome

A custom C++ cache proxy reduced free-tier LLM API calls by 73% in a simulated CI workload, cutting median latency from 1.8s to 12ms for repeated requests and extending quota utility by 3.7x.

Workflow

• Design cache key: Normalize JSON body by removing non-deterministic fields (timestamps, nonces) and hash with SHA-256.
• Store responses in SQLite: Use a simple table with key, response, and created_at columns for ACID compliance and single-file storage.
• Implement proxy logic: Listen on a local port, check cache for hits; on miss, forward to upstream, buffer response, store in DB, and return.
• Handle streaming: Disable streaming (set stream: false) for cacheable requests to allow full buffering before caching.
• Add eviction policy: Implement a fixed-size LRU cache (e.g., 1,000 entries) backed by SQLite to manage disk space and lookup speed.
• Set TTL: Use a short TTL (e.g., 30 seconds) to balance freshness and hit rate for bursty CI workloads.

Evidence

In a simulated CI workload, it cut model calls from 1,000 to 270. That is a 73% reduction

Source

Median latency: 1.8s (Without cache) vs 12ms (With cache)

Source