Building Amit.AI, Part 1: The Frontend

Описание и применение

Быстрый клиентский ответ на FAQ, отложенный вызов LLM, потоковая обработка токенов через fetch, предварительный запрос для пробуждения сервера, безопасный рендеринг.

Задача

Provide instant, free FAQ answers in a browser while deferring to an LLM only when necessary, avoiding free‑tier quota waste and backend latency.

Результат

Instant FAQ answers, reduced LLM usage, mitigated backend sleep latency, and safe streaming rendering.

Рабочий процесс

• 1. Detect if user message is confident using a token‑overlap matcher with thresholds and stopword filtering.
• 2. If confident, return FAQ answer immediately.
• 3. If not confident, send message to backend via POST and stream tokens using a hand‑rolled fetch reader.
• 4. Render tokens incrementally, keeping typing indicator until first token.
• 5. Warm backend with a health‑endpoint ping during boot animation.
• 6. Apply guardrails to block jailbreak‑style or disallowed content before any answer is produced.

Доказательства

'Most questions never reach an LLM. A tiny client-side matcher answers them instantly, and when it is unsure, it stays quiet and defers.'

Источник

The matcher returns one of three things: guard, faq or defer. There is deliberately no fourth "I'm not sure" answer. A defer is silent.

Источник

SSE frames are separated by a blank line, and a network chunk can end in the middle of one. So you buffer, split on \n\n, and keep the leftover for the next read.

Источник