How I Built an OTel‑Native GPU Profiler & LLM Gateway to Stop Bleeding Inference Costs

Description and use

An async FastAPI gateway wraps LLM calls with OpenTelemetry, profiles CUDA memory, calculates per‑token cost, and exports metrics to Prometheus/Grafana, enabling precise GPU cost control.

Problem

Running LLM workloads at scale creates a blind spot: opaque GPU utilization and runaway inference costs, leading to over‑provisioned expensive hardware.

Outcome

The gateway provides real‑time GPU telemetry and per‑token cost, revealing that many requests only need an L4 GPU, reducing cloud spend by avoiding unnecessary A100 rentals.

Workflow

• 1. n8n triggers HTTP request to gateway with lead data
• 2. Gateway sanitizes domain and builds structured prompt
• 3. Gateway records pre‑inference VRAM and starts latency timer
• 4. Gateway performs LLM inference (e.g., vLLM)
• 5. Gateway calculates per‑token cost and records post‑inference metrics
• 6. Gateway returns inference result and cost metadata to n8n
• 7. Metrics exported to Prometheus and visualized in Grafana

Evidence

Traditional cloud monitoring lacks granular, step-by-step visibility into CUDA kernel execution and per-token generation costs.

Source

We will use this async middleware to sit between an n8n lead enrichment pipeline and your custom AI backend, tracking real-time CUDA memory, calculating per-token inference costs, and exporting metrics natively to Prometheus and Grafana.

Source