My demo refused 96% of off-topic questions but only 39% of tricky ones. I know because of a test set
Описание и применение
<p>Here are two numbers from my document Q&A demo. Ask it about something that is not in the knowledge base at all, and it refuses correctly 96.1% of the time. Ask it a question that sits close to a document but is not answered by it, and it refuses only 39% of the time.</p> <p>If I had tried the demo by hand, I would have asked a few questions, seen it say "not in the docs" a couple of times, and called it good. The 39% only showed up because there was a labelled test set. That set is the subject of this post: a <strong>golden set</strong>, the cheapest acceptance test an AI feature can h
Задача
<p>Here are two numbers from my document Q&A demo. Ask it about something that is not in the knowledge base at all, and it refuses correctly 96.1% of the time. Ask it a question that sits close to a document but is not answered by it, and it refuses only 39% of the time.</p> <p>If I had tried the demo by hand, I would have asked a few questions, seen it say "not in the docs" a couple of times, and called it good. The 39% only showed up because there was a labelled test set. That set is the subject of this post: a <strong>golden set</strong>, the cheapest acceptance test an AI feature can h
Результат
<p>Here are two numbers from my document Q&A demo. Ask it about something that is not in the knowledge base at all, and it refuses correctly 96.1% of the time. Ask it a question that sits close to a document but is not answered by it, and it refuses only 39% of the time.</p> <p>If I had tried the demo by hand, I would have asked a few questions, seen it say "not in the docs" a couple of times, and called it good. The 39% only showed up because there was a labelled test set. That set is the subject of this post: a <strong>golden set</strong>, the cheapest acceptance test an AI feature can h
Доказательства
My demo refused 96% of off-topic questions but only 39% of tricky ones. I know because of a test set
- Дата записи: 2026-11-01T06:13:48.052Z