Why We Built an AI Gateway in Go: Failover, PII Redaction, and Sub-Millisecond Caching

Description and use

SentinelGateway is a Go-based proxy that provides failover, in-flight PII scrubbing, and semantic caching, reducing downtime, compliance risks, and token costs for production LLM deployments.

Problem

Teams deploying LLMs face upstream downtime, compliance data leaks, and runaway token costs.

Outcome

Teams using SentinelGateway experienced reduced application crashes from provider outages, eliminated GDPR/SOC2 data leakage risks, and cut token costs by caching repeated prompts.

Workflow

• 1. Install SentinelGateway and set base_url to https://sentinelgateway.ai/v1 in the OpenAI client.
• 2. Configure primary and secondary providers in SentinelGateway’s Redis-backed health monitor.
• 3. Enable PII scrubbing by default; the gateway scans and redacts SSNs, credit cards, emails, and API keys before sending to upstream.
• 4. Activate semantic caching; SentinelGateway hashes prompts and checks Redis for cache hits, returning cached responses in <15 ms.
• 5. Deploy the gateway behind your application; it handles failover automatically when a provider returns 429 or 503.
• 6. Monitor usage and quotas via SentinelGateway’s Lua scripts that atomically decrement counters across nodes.

Evidence

SentinelGateway implements the complete /v1/chat/completions schema.

Source

High-speed token scanners detect SSNs, credit card numbers, API keys, and email addresses.

Source

Cache hits return in sub-15ms with zero upstream token consumption.

Source