How I Built an OTel‑Native GPU Profiler & LLM Gateway to Stop Bleeding Inference Costs

Описание и применение

Создана асинхронная шлюзовая служба на FastAPI, оборачивающая вызовы LLM с OpenTelemetry, профилирующая CUDA‑память и рассчитывающая стоимость по токенам, что позволяет точнее управлять расходами на GPU.

Задача

Running LLM workloads at scale creates a blind spot: opaque GPU utilization and runaway inference costs, leading to over‑provisioned expensive hardware.

Результат

The gateway provides real‑time GPU telemetry and per‑token cost, revealing that many requests only need an L4 GPU, reducing cloud spend by avoiding unnecessary A100 rentals.

Рабочий процесс

• 1. n8n triggers HTTP request to gateway with lead data
• 2. Gateway sanitizes domain and builds structured prompt
• 3. Gateway records pre‑inference VRAM and starts latency timer
• 4. Gateway performs LLM inference (e.g., vLLM)
• 5. Gateway calculates per‑token cost and records post‑inference metrics
• 6. Gateway returns inference result and cost metadata to n8n
• 7. Metrics exported to Prometheus and visualized in Grafana

Доказательства

Traditional cloud monitoring lacks granular, step-by-step visibility into CUDA kernel execution and per-token generation costs.

Источник

We will use this async middleware to sit between an n8n lead enrichment pipeline and your custom AI backend, tracking real-time CUDA memory, calculating per-token inference costs, and exporting metrics natively to Prometheus and Grafana.

Источник