Sulci 1.0: a context‑aware semantic cache for agents, and the number most caches don’t publish

说明与用途

Sulci 1.0 是一种面向 LLM 代理的上下文感知语义缓存，可将重复模型调用降低至 99.7%，缓存命中率 93.8%，延迟 87 ms。

问题

LLM agents generate many near‑duplicate queries, causing repeated expensive model calls.

结果

Sulci 1.0 demonstrates that a lightweight semantic cache can reduce LLM dispatches by up to 99.7% in warm sessions, with 93.8% of cache hits correct and 98.9% correct declines when no answer is cached. Cache hit latency averages 87 ms, dominated by embedding time.

工作流

• Clone the Sulci repo and run the benchmark to verify performance.
• Install the library with pip: pip install "sulci[sqlite]". Create a Cache instance: cache = Cache(backend="sqlite", threshold=0.85). For each query: call cache.get(query); if no response, call the LLM and then cache.set(query, response). Optionally pass session_id to enable context‑aware blending. Integrate with existing stacks via one‑line adapters (LangChain, LlamaIndex, LiteLLM, proxy, MCP, TypeScript).

证据

With a cold, empty cache, 33.4% [30.00–35.00%] of dispatches could be served from cache.

来源

Warm, within the same session, 99.7% [99.69–99.81%].

来源

Of cache hits, share that are correct 93.8% [92.13–94.69%].

来源

Cache hit, p50 87ms

来源

pip install "sulci[sqlite]"

来源