Self-hosted Langfuse: tracing 7% of my AI agents, and ClickHouse logging itself

Описание и применение

Автор самоустанавливает Langfuse для отслеживания AI‑агентов, но изначально включил трассировку только для одного профиля, что привело к покрытию 7 % и чрезмерному использованию диска ClickHouse. После настройки ключей и тегов для всех профилей и отключения системных логов ClickHouse, покрытие стало 100 % и дисковое пространство сократилось почти вдвое.

Задача

Only 7% of AI agent traffic was traced and ClickHouse consumed 6 GiB of disk for 2.2 MiB of trace data, obscuring performance insights and wasting resources.

Результат

Tracing coverage increased from 7 % to 100 % across profiles, enabling accurate latency and cost analysis per role. ClickHouse disk usage dropped by ~50 % after disabling unnecessary system logs, improving I/O performance.

Рабочий процесс

• 1. Copy Langfuse API keys into each profile’s secret scope. 2. Enable the Langfuse tracing plugin for each profile. 3. Set the Langfuse environment to the profile name. 4. Restart the gateway and send a probe trace from each profile to verify tagging. 5. Create a ClickHouse config.d file that removes twelve system log tables and sets logger level to warning. 6. Mount the config file read‑only into the container and recreate it. 7. Verify container health, unchanged trace_log row count, and intact trace data.
• 8. Run a weekly script that compares session records with Langfuse traces to generate role‑based reports.

Доказательства

Langfuse held 618 events across 79 traces in four weeks. That's about three traces a day.

Источник

ClickHouse was using 6.03 GiB of disk. The actual Langfuse trace data in it was 2.2 MiB.

Источник

The tracing plugin is enabled per profile, and it reads its Langfuse API keys through that profile's own secret scope.

Источник

ClickHouse lets you override server config with a file dropped into /etc/clickhouse-server/config.d/. This one removes twelve system log tables...

Источник