Claude's Context Window in Practice: Caching, Chunking, and Cost

Description and use

Choosing how to provide Claude with large documents: count tokens, cache the whole text, or chunk and retrieve relevant parts, with minimal Python code and cost estimation.

Problem

Choosing how to provide Claude with large documents—whether to embed the whole text, cache it, or retrieve relevant chunks—while balancing cost and performance.

Outcome

By counting tokens first, users can decide whether to cache the entire document or retrieve relevant chunks. Caching reduces repeated token costs for stable content, while chunking and retrieval keep costs low for large or frequently changing corpora.

Workflow

• Count tokens of the full document with Anthropic SDK to assess fit within context window.
• If token count is small relative to context window, send entire document and enable caching via cache_control in the system message.
• If token count is large or document changes frequently, split the text into overlapping paragraph chunks using the provided chunk function.
• Embed each chunk with a chosen embedding model and compute similarity scores to the user’s question.
• Select top-k relevant chunks and assemble them into a prompt for Claude.
• Send the prompt to Claude and receive the answer.
• Monitor usage fields such as cache_read_input_tokens to verify caching effectiveness.
• Adjust chunk size, overlap, or k as needed based on performance and cost.

Evidence

The SDK can count tokens for a request without running it

Source

If the number is a small fraction of the model's context window, long context is on the table

Source

Three habits make caching work: Put stable content first and changing content last

Source

When the corpus is large, changes often, or has to respect per-user permissions, retrieval is the better fit

Source