Building Amit.AI, Part 1: The Frontend

说明与用途

快速客户端 FAQ 回答，延迟 LLM 调用，使用 fetch 进行令牌流式处理，后端预热请求，安全渲染。

问题

Provide instant, free FAQ answers in a browser while deferring to an LLM only when necessary, avoiding free‑tier quota waste and backend latency.

结果

Instant FAQ answers, reduced LLM usage, mitigated backend sleep latency, and safe streaming rendering.

工作流

• 1. Detect if user message is confident using a token‑overlap matcher with thresholds and stopword filtering.
• 2. If confident, return FAQ answer immediately.
• 3. If not confident, send message to backend via POST and stream tokens using a hand‑rolled fetch reader.
• 4. Render tokens incrementally, keeping typing indicator until first token.
• 5. Warm backend with a health‑endpoint ping during boot animation.
• 6. Apply guardrails to block jailbreak‑style or disallowed content before any answer is produced.

证据

'Most questions never reach an LLM. A tiny client-side matcher answers them instantly, and when it is unsure, it stays quiet and defers.'

来源

The matcher returns one of three things: guard, faq or defer. There is deliberately no fourth "I'm not sure" answer. A defer is silent.

来源

SSE frames are separated by a blank line, and a network chunk can end in the middle of one. So you buffer, split on \n\n, and keep the leftover for the next read.

来源