Semantic Caching for AI Agents in Production

Description and use

Semantic caching for AI agents on AWS cuts LLM costs by returning stored answers on cache hits. Choose DynamoDB for serverless scalability or Valkey for low‑latency in‑memory lookups based on traffic patterns.

Problem

AI agents repeatedly pay full LLM cost for questions already answered, wasting compute and money.

Outcome

Semantic caching reduces LLM token usage to zero on cache hits, cutting costs by the cost of embedding and lookup while maintaining response freshness via TTL. In-memory Valkey offers faster misses; DynamoDB provides serverless, scalable storage.

Workflow

• 1. Embed incoming question using Amazon Titan Text Embeddings V2.
• 2. Search vector index (DynamoDB or Valkey) for nearest match.
• 3. If similarity > threshold and exact metadata match, return cached answer (hit).
• 4. If not, run agent: invoke Bedrock LLM, execute tools, generate answer.
• 5. Store new question, answer, metadata, and TTL in chosen cache store.
• 6. Repeat for subsequent queries.
• 7. Optionally use reasoning or plan template caches to reuse intermediate steps.

Evidence

A word‑for‑word repeat costs 0 LLM tokens unless a guard below has to rewrite the stored answer.

Source

The write side is eventually consistent, which matters when the index is a cache.

Source