If your scorer can't return zero, it will rank a dead thread

Description and use

<p>Most agent systems eventually grow a ranked worklist. Leads, alerts, retries, documents to review, tickets to triage. You write a scoring function, sort descending, show the top ten, and hand the list to a human or to a model that acts on it.</p> <p>The bug that shows up in every one of these is the same: the score measures how <em>interesting</em> a row is, and nobody ever checks whether the row can be acted on at all. So the list fills with the most interesting things you cannot do anything about, and because the list is sorted, they go on top.</p> <p>I built one of these last month and i

Problem

<p>Most agent systems eventually grow a ranked worklist. Leads, alerts, retries, documents to review, tickets to triage. You write a scoring function, sort descending, show the top ten, and hand the list to a human or to a model that acts on it.</p> <p>The bug that shows up in every one of these is the same: the score measures how <em>interesting</em> a row is, and nobody ever checks whether the row can be acted on at all. So the list fills with the most interesting things you cannot do anything about, and because the list is sorted, they go on top.</p> <p>I built one of these last month and i

Outcome

<p>Most agent systems eventually grow a ranked worklist. Leads, alerts, retries, documents to review, tickets to triage. You write a scoring function, sort descending, show the top ten, and hand the list to a human or to a model that acts on it.</p> <p>The bug that shows up in every one of these is the same: the score measures how <em>interesting</em> a row is, and nobody ever checks whether the row can be acted on at all. So the list fills with the most interesting things you cannot do anything about, and because the list is sorted, they go on top.</p> <p>I built one of these last month and i

Evidence

If your scorer can't return zero, it will rank a dead thread

Source