I Thought Building Agent Observability Was a Detector Problem. I Was Wrong.

Описание и применение

Автор создал инструмент наблюдаемости для ИИ‑агентов, но столкнулся с тем, что реальные трассы отличаются от синтетических. Он научился нормализовать данные, использовать синтетические трассы для проверки детекторов и проводить строгие интеграционные тесты, чтобы сделать систему надёжной.

Задача

Traditional observability cannot pinpoint why an AI agent behaves poorly—e.g., repeated tool calls, hallucinations, or token waste—making it hard to diagnose and fix agent run failures.

Результат

The author learned that real trace data shape, early field testing, and realistic validation are as important as detector logic. The project now includes robust normalization, synthetic trace generation, and stricter end‑to‑end gates, improving detector reliability and observability usefulness.

Рабочий процесс

• 1. Instrument agent runs with AgentTracer and emit spans for planning, tool calls, etc.
• 2. Run rule‑based detectors on the generated traces.
• 3. Test on synthetic traces to ensure detectors can see required fields.
• 4. Validate on real Hugging Face traces, performing normalization passes for inconsistent keys and timestamps.
• 5. Iterate on detector thresholds and trace schema based on real‑world feedback.
• 6. Implement end‑to‑end smoke tests to confirm OTLP export, Jaeger ingestion, and API serving.
• 7. Publish OSS with PyPI packages and a comprehensive test suite.

Доказательства

On my first large pass over 100,000 real agent traces from Hugging Face, my empty_response detector fired on 100% of them. Every trace.

Источник

The synthetic trace generator produced one million traces, and structural compatibility jumped to 99.2%.

Источник

I had marked the OTLP export milestone done. The gate was green. The docs looked fine. The demo looked fine.

Источник