Why We Built an AI Gateway in Go: Failover, PII Redaction, and Sub-Millisecond Caching

Описание и применение

SentinelGateway – это прокси на Go, обеспечивающий отказоустойчивость, удаление PII и кэширование запросов, что снижает сбои, риски утечки данных и расходы на токены.

Задача

Teams deploying LLMs face upstream downtime, compliance data leaks, and runaway token costs.

Результат

Teams using SentinelGateway experienced reduced application crashes from provider outages, eliminated GDPR/SOC2 data leakage risks, and cut token costs by caching repeated prompts.

Рабочий процесс

• 1. Install SentinelGateway and set base_url to https://sentinelgateway.ai/v1 in the OpenAI client.
• 2. Configure primary and secondary providers in SentinelGateway’s Redis-backed health monitor.
• 3. Enable PII scrubbing by default; the gateway scans and redacts SSNs, credit cards, emails, and API keys before sending to upstream.
• 4. Activate semantic caching; SentinelGateway hashes prompts and checks Redis for cache hits, returning cached responses in <15 ms.
• 5. Deploy the gateway behind your application; it handles failover automatically when a provider returns 429 or 503.
• 6. Monitor usage and quotas via SentinelGateway’s Lua scripts that atomically decrement counters across nodes.

Доказательства

SentinelGateway implements the complete /v1/chat/completions schema.

Источник

High-speed token scanners detect SSNs, credit card numbers, API keys, and email addresses.

Источник

Cache hits return in sub-15ms with zero upstream token consumption.

Источник