Semantic Caching for AI Agents in Production

Описание и применение

Семантическое кэширование для AI‑агентов на AWS снижает расходы на LLM, возвращая сохранённые ответы при совпадении вопросов. Выбор между DynamoDB и Valkey зависит от нагрузки и требований к латентности.

Задача

AI agents repeatedly pay full LLM cost for questions already answered, wasting compute and money.

Результат

Semantic caching reduces LLM token usage to zero on cache hits, cutting costs by the cost of embedding and lookup while maintaining response freshness via TTL. In-memory Valkey offers faster misses; DynamoDB provides serverless, scalable storage.

Рабочий процесс

• 1. Embed incoming question using Amazon Titan Text Embeddings V2.
• 2. Search vector index (DynamoDB or Valkey) for nearest match.
• 3. If similarity > threshold and exact metadata match, return cached answer (hit).
• 4. If not, run agent: invoke Bedrock LLM, execute tools, generate answer.
• 5. Store new question, answer, metadata, and TTL in chosen cache store.
• 6. Repeat for subsequent queries.
• 7. Optionally use reasoning or plan template caches to reuse intermediate steps.

Доказательства

A word‑for‑word repeat costs 0 LLM tokens unless a guard below has to rewrite the stored answer.

Источник

The write side is eventually consistent, which matters when the index is a cache.

Источник