Node.js Moderation LLM API Bill: 5 SaaS Small-Model Routing and Batch Checks

Description and use

Implemented a cheap‑model‑first routing strategy with strict JSON schema validation and confidence thresholds, reducing LLM API costs and engineering overhead.

Problem

High LLM API costs in a Node.js SaaS moderation app due to counting raw calls, malformed JSON, unnecessary large-model fallbacks, delayed work, and engineering effort reconciling usage across providers.

Outcome

Implemented a cheap‑model‑first routing strategy with strict schema validation and confidence thresholds, reducing LLM API costs by avoiding unnecessary large‑model calls and batching low‑confidence work.

Workflow

• 1. Define a strict moderation decision schema (label, confidence, needsHumanReview, reason).
• 2. Route each report to a cheap model (e.g., deepseek‑v4‑flash).
• 3. Validate the response against the schema; if it passes and confidence exceeds a threshold, accept the decision.
• 4. If validation fails or confidence is below the threshold, fallback to a larger model (e.g., deepseek‑v4‑pro).
• 5. If fallback also fails or confidence remains low, flag for human review or batch for later processing.
• 6. Use Infrai’s API to retrieve per‑call cost, vendor, and latency metadata for cost comparison.
• 7. Record metrics (schema‑pass rate, fallback rate, token usage, cost per accepted classification) to tune thresholds.
• 8. Batch non‑urgent work that has no user waiting to reduce per‑call overhead.

Evidence

Send routine reports to a smaller model first, but promote a report when the output fails validation or the confidence falls below a threshold chosen by the product team.

Source

Infrai exposes per‑call cost, vendor, and latency metadata on its native and OpenAI‑compatible surfaces.

Source