Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch

Описание и применение

Легкая двухступенчатая схема обработки логов домашней автоматизации: Fluent Bit распознаёт и предварительно обрабатывает различные форматы, а Elasticsearch завершает декодирование JSON и хранит чистые документы для визуализации в Kibana и Grafana.

Задача

Home automation systems generate diverse log formats that are hard to ingest into a unified analytics stack.

Результат

A lightweight two‑stage pipeline that keeps Fluent Bit configuration simple while leveraging Elasticsearch’s ingest pipelines for final JSON decoding, achieving sub‑minute ingestion latency and clean, queryable data.

Рабочий процесс

• 1. Configure Fluent Bit to tail each log source with a unique tag and persistent DB.
• 2. Define multiple small parsers for recurring Domoticz formats and generic JSON records in fluent‑bit/parsers.conf.
• 3. Apply the parser filter to each tag, reserving original data and adding parsed fields.
• 4. Remove intermediate fields (e.g., msgtxt) with record_modifier filter.
• 5. Forward preprocessed records to Elasticsearch via the es output plugin, including tag key for source identification.
• 6. Create an Elasticsearch ingest pipeline that decodes any remaining JSON in the log field and promotes its contents to the document root.
• 7. Index documents into dom_v3 and visualize with Kibana/Grafana.
• 8. Monitor pipeline latency (typically <1 min) and adjust parsers or pipeline processors as new log formats appear.

Доказательства

The complete pipeline normally has a latency of less than one minute in the author's installation.

Источник

Fluent Bit performs the first level of processing: reading the various log files; recognizing several recurring line formats; extracting timestamps and metadata; decoding JSON when it is directly available; removing intermediate fields; forwarding the resulting records to Elasticsearch.

Источник