Migrating 35KB Prompts from Opus to Ollama: Key Gotchas

Description and use

Migrating a 35 KB prompt from Opus to Ollama caused truncation due to token limits and altered output quality due to different system message handling. The fix involved rewriting the prompt, adjusting the request, adding caching, and implementing error handling.

Problem

A 35 KB pre‑prompt from Opus exceeded Ollama’s default token limit, causing truncation and unpredictable model behavior, and Ollama’s different handling of system messages altered output quality.

Outcome

Prompt successfully migrated to Ollama with token limits respected, output quality restored, and latency reduced through caching.

Workflow

• Audit prompt length with a matching tokenizer
• Rewrite prompt to stay within Ollama’s token limit
• Adjust request payload format for Ollama’s system message handling
• Add a caching layer (e.g., Redis) to store the prompt payload
• Wrap LLM call in try‑catch to handle truncation errors
• Implement graceful fallback or logging for token limit errors

Evidence

The prompt was larger than Ollama’s default token limit, so the request was truncated and the model behaved unpredictably.

Source

Ollama treats system messages and prompt formatting differently, which altered output quality.

Source