Why Splitting Logs by Business Key Becomes a Resource-Management Problem

Описание и применение

Разделение логов по бизнес‑ключу облегчает трассировку, но при высокой кардинальности приводит к исчерпанию дескрипторов файлов и падению производительности. Решение – использовать Log4Key с ограничением открытых файлов, LRU‑кешем и пакетными записями.

Задача

When logs are split by business key (player, order, tenant), high cardinality leads to file descriptor exhaustion and performance degradation.

Результат

Log4Key allows per‑business‑object log files, but high key cardinality triggers file descriptor limits, leading to event rejections and throughput drops. The solution requires careful tuning of maxOpenFiles, worker threads, and batching.

Рабочий процесс

• 1. Add a business key to each log event using Log4KeyLogger.
• 2. Configure Log4Key to route events to files named after the key.
• 3. Set maxOpenFiles to cap open file descriptors.
• 4. Use LRU policy to close idle files and open new ones.
• 5. Batch writes and flush periodically to reduce I/O overhead.
• 6. Monitor for rejections and adjust worker count or maxOpenFiles as needed.

Доказательства

At 200 keys there's no rejection. At 210, rejection shows up — mailbox full, events dropped — and write throughput falls off noticeably.

Источник

None of this changes what the disk can actually sustain. What it does is keep resource exhaustion contained within the logging layer, instead of letting it take down the app.

Источник