How I Built an OTel‑Native GPU Profiler & LLM Gateway to Stop Bleeding Inference Costs

说明与用途

使用 FastAPI 构建异步网关，结合 OpenTelemetry 进行 CUDA 内存监控和按 token 计费，输出 Prometheus/Grafana 指标，实现 GPU 成本精准控制。

问题

Running LLM workloads at scale creates a blind spot: opaque GPU utilization and runaway inference costs, leading to over‑provisioned expensive hardware.

结果

The gateway provides real‑time GPU telemetry and per‑token cost, revealing that many requests only need an L4 GPU, reducing cloud spend by avoiding unnecessary A100 rentals.

工作流

• 1. n8n triggers HTTP request to gateway with lead data
• 2. Gateway sanitizes domain and builds structured prompt
• 3. Gateway records pre‑inference VRAM and starts latency timer
• 4. Gateway performs LLM inference (e.g., vLLM)
• 5. Gateway calculates per‑token cost and records post‑inference metrics
• 6. Gateway returns inference result and cost metadata to n8n
• 7. Metrics exported to Prometheus and visualized in Grafana

证据

Traditional cloud monitoring lacks granular, step-by-step visibility into CUDA kernel execution and per-token generation costs.

来源

We will use this async middleware to sit between an n8n lead enrichment pipeline and your custom AI backend, tracking real-time CUDA memory, calculating per-token inference costs, and exporting metrics natively to Prometheus and Grafana.

来源