Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch

Description and use

A lightweight two‑stage pipeline that uses Fluent Bit to parse heterogeneous IoT logs and Elasticsearch ingest pipelines to finalize JSON decoding, enabling fast, clean analytics in Kibana and Grafana.

Problem

Home automation systems generate diverse log formats that are hard to ingest into a unified analytics stack.

Outcome

A lightweight two‑stage pipeline that keeps Fluent Bit configuration simple while leveraging Elasticsearch’s ingest pipelines for final JSON decoding, achieving sub‑minute ingestion latency and clean, queryable data.

Workflow

• 1. Configure Fluent Bit to tail each log source with a unique tag and persistent DB.
• 2. Define multiple small parsers for recurring Domoticz formats and generic JSON records in fluent‑bit/parsers.conf.
• 3. Apply the parser filter to each tag, reserving original data and adding parsed fields.
• 4. Remove intermediate fields (e.g., msgtxt) with record_modifier filter.
• 5. Forward preprocessed records to Elasticsearch via the es output plugin, including tag key for source identification.
• 6. Create an Elasticsearch ingest pipeline that decodes any remaining JSON in the log field and promotes its contents to the document root.
• 7. Index documents into dom_v3 and visualize with Kibana/Grafana.
• 8. Monitor pipeline latency (typically <1 min) and adjust parsers or pipeline processors as new log formats appear.

Evidence

The complete pipeline normally has a latency of less than one minute in the author's installation.

Source

Fluent Bit performs the first level of processing: reading the various log files; recognizing several recurring line formats; extracting timestamps and metadata; decoding JSON when it is directly available; removing intermediate fields; forwarding the resulting records to Elasticsearch.

Source