Semantic Caching for AI Agents in Production

说明与用途

在 AWS 上为 AI 代理实现语义缓存，通过在缓存命中时返回已存答案来降低 LLM 成本。根据流量模式选择 DynamoDB（无服务器）或 Valkey（低延迟内存）存储。

问题

AI agents repeatedly pay full LLM cost for questions already answered, wasting compute and money.

结果

Semantic caching reduces LLM token usage to zero on cache hits, cutting costs by the cost of embedding and lookup while maintaining response freshness via TTL. In-memory Valkey offers faster misses; DynamoDB provides serverless, scalable storage.

工作流

• 1. Embed incoming question using Amazon Titan Text Embeddings V2.
• 2. Search vector index (DynamoDB or Valkey) for nearest match.
• 3. If similarity > threshold and exact metadata match, return cached answer (hit).
• 4. If not, run agent: invoke Bedrock LLM, execute tools, generate answer.
• 5. Store new question, answer, metadata, and TTL in chosen cache store.
• 6. Repeat for subsequent queries.
• 7. Optionally use reasoning or plan template caches to reuse intermediate steps.

证据

A word‑for‑word repeat costs 0 LLM tokens unless a guard below has to rewrite the stored answer.

来源

The write side is eventually consistent, which matters when the index is a cache.

来源