Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack

Description and use

An OpenTelemetry instrumentation library (otel-swarm) for multi-agent LLM systems that tracks roles, model fallbacks, and review gates without corrupting aggregated metrics, complete with SigNoz dashboards.

Problem

Multi-agent LLM systems execute concurrent tasks across multiple models with flat, interleaved logs, making it difficult to trace parent-child structures, pinpoint which agent caused latency or cost overruns, or track silent fallback promotions and failing review gates.

Outcome

Extracted multi-agent instrumentation into a lightweight MIT library (otel-swarm) that successfully traces hundreds of model calls and tokens across complex swarms while maintaining accurate grouping attributes and visibility into fallback promotions and review-gate failures.

Workflow

• Install the otel-swarm library via npm from GitHub.
• Initialize the swarm with createSwarm() providing service name and OTLP endpoint.
• Wrap generation tasks using swarm.task(), agent roles using swarm.agent(), and model calls using swarm.llm().
• Record fallbacks and critic events as span events rather than overwriting aggregated attributes.
• Export spans to SigNoz or console and query events in ClickHouse using ARRAY JOIN or arrayExists.
• Import pre-built SigNoz dashboards and alert rules for monitoring.

Evidence

Running npm run example against a local SigNoz produced 9 spans under service otel-swarm-demo: a root generation span of 1325ms, then agent.planner/llm.planner...

Source

In production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.

Source