Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack

说明与用途

用于多智能体LLM系统的OpenTelemetry插桩库（otel-swarm），可跟踪角色、模型回退和审查关卡而不损坏聚合指标，并配有SigNoz仪表板。

问题

Multi-agent LLM systems execute concurrent tasks across multiple models with flat, interleaved logs, making it difficult to trace parent-child structures, pinpoint which agent caused latency or cost overruns, or track silent fallback promotions and failing review gates.

结果

Extracted multi-agent instrumentation into a lightweight MIT library (otel-swarm) that successfully traces hundreds of model calls and tokens across complex swarms while maintaining accurate grouping attributes and visibility into fallback promotions and review-gate failures.

工作流

• Install the otel-swarm library via npm from GitHub.
• Initialize the swarm with createSwarm() providing service name and OTLP endpoint.
• Wrap generation tasks using swarm.task(), agent roles using swarm.agent(), and model calls using swarm.llm().
• Record fallbacks and critic events as span events rather than overwriting aggregated attributes.
• Export spans to SigNoz or console and query events in ClickHouse using ARRAY JOIN or arrayExists.
• Import pre-built SigNoz dashboards and alert rules for monitoring.

证据

Running npm run example against a local SigNoz produced 9 spans under service otel-swarm-demo: a root generation span of 1325ms, then agent.planner/llm.planner...

来源

In production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.

来源