Self-hosted Langfuse: tracing 7% of my AI agents, and ClickHouse logging itself

Описание и применение

Автор самостоял Langfuse для отслеживания AI‑агентов, но из‑за неверной настройки только 7 % запросов были записаны, а ClickHouse занимал 6 ГБ диска, в основном из‑за системных логов. После включения трассировки для всех профилей и отключения ненужных таблиц ClickHouse, покрытие выросло до 100 %, а дисковое пространство сократилось почти вдвое.

Задача

Only 7% of AI agent calls were traced and ClickHouse consumed 6 GiB of disk for 2.2 MiB of trace data, obscuring real usage and inflating storage costs.

Результат

Tracing coverage increased from 7% to 100% across profiles, enabling accurate cost and latency analysis. Disk usage dropped by ~50% by disabling ClickHouse system logs, while trace data remained intact.

Рабочий процесс

• 1. Copy Langfuse API keys into each profile’s env file. 2. Enable the Langfuse tracing plugin for each profile. 3. Set the Langfuse environment to the profile name. 4. Restart the gateway and send a probe trace from each profile to verify tagging. 5. Create a ClickHouse config.d file that removes unnecessary system log tables and sets logger level to warning. 6. Mount the config file read‑only into the container and recreate the container. 7. Verify container health, unchanged trace_log row count, and intact trace data. 8. Run a weekly script that aggregates session records and Langfuse traces to generate reports.

Доказательства

Langfuse held 618 events across 79 traces in four weeks. That's about three traces a day.

Источник

ClickHouse was using 6.03 GiB of disk. The actual Langfuse trace data in it was 2.2 MiB.

Источник

The third step is the one that matters for later. Traces carry no profile tag of their own, so without it every trace lands in one undifferentiated pile.

Источник

ClickHouse lets you override server config with a file dropped into /etc/clickhouse-server/config.d/. This one removes twelve system log tables...

Источник