A 200 OK Can Still Be a Broken Answer: Check Why Your LLM Stopped

Description and use

<p>Here's a bug that never shows up in your error dashboard: the LLM call succeeds, returns HTTP 200, your code takes the text and moves on... and the text is half an answer. Or half a JSON object. Or a polite refusal your app renders as if it were the result.</p> <p>Nothing threw. Nothing retried. Your monitoring says everything is green.</p> <p>The fix is one field that every major LLM API already gives you, and that a lot of code never reads: <strong>the reason the model stopped generating.</strong></p> <p>(This morning's post was about RAG chunking, what goes <em>into</em> the model. This

Problem

<p>Here's a bug that never shows up in your error dashboard: the LLM call succeeds, returns HTTP 200, your code takes the text and moves on... and the text is half an answer. Or half a JSON object. Or a polite refusal your app renders as if it were the result.</p> <p>Nothing threw. Nothing retried. Your monitoring says everything is green.</p> <p>The fix is one field that every major LLM API already gives you, and that a lot of code never reads: <strong>the reason the model stopped generating.</strong></p> <p>(This morning's post was about RAG chunking, what goes <em>into</em> the model. This

Outcome

<p>Here's a bug that never shows up in your error dashboard: the LLM call succeeds, returns HTTP 200, your code takes the text and moves on... and the text is half an answer. Or half a JSON object. Or a polite refusal your app renders as if it were the result.</p> <p>Nothing threw. Nothing retried. Your monitoring says everything is green.</p> <p>The fix is one field that every major LLM API already gives you, and that a lot of code never reads: <strong>the reason the model stopped generating.</strong></p> <p>(This morning's post was about RAG chunking, what goes <em>into</em> the model. This

Evidence

A 200 OK Can Still Be a Broken Answer: Check Why Your LLM Stopped

Source