Why Splitting Logs by Business Key Becomes a Resource-Management Problem

说明与用途

按业务键拆分日志简化追踪，但高基数会耗尽文件描述符并降低性能。使用 Log4Key 并设置 maxOpenFiles、LRU 缓存和批量写入来控制资源使用。

问题

When logs are split by business key (player, order, tenant), high cardinality leads to file descriptor exhaustion and performance degradation.

结果

Log4Key allows per‑business‑object log files, but high key cardinality triggers file descriptor limits, leading to event rejections and throughput drops. The solution requires careful tuning of maxOpenFiles, worker threads, and batching.

工作流

• 1. Add a business key to each log event using Log4KeyLogger.
• 2. Configure Log4Key to route events to files named after the key.
• 3. Set maxOpenFiles to cap open file descriptors.
• 4. Use LRU policy to close idle files and open new ones.
• 5. Batch writes and flush periodically to reduce I/O overhead.
• 6. Monitor for rejections and adjust worker count or maxOpenFiles as needed.

证据

At 200 keys there's no rejection. At 210, rejection shows up — mailbox full, events dropped — and write throughput falls off noticeably.

来源

None of this changes what the disk can actually sustain. What it does is keep resource exhaustion contained within the logging layer, instead of letting it take down the app.

来源