Optimizing Disk I/O in NumPy with LZ4 C-Extensions

说明与用途

本文介绍了 numpy-cache 库，它使用 C 扩展和 LZ4 算法来快速缓存 NumPy 数组。该解决方案消除了写入速度和文件大小之间的权衡，提供了接近未压缩保存的性能，但压缩效果优于标准的 zlib。

问题

Developers face a trade-off between disk I/O speed and storage space when handling large NumPy arrays. Standard np.save() is fast but uncompressed, while np.savez_compressed() is slow due to single-threaded DEFLATE, and HDF5/Zarr introduce heavy dependencies.

结果

The implementation provides a lightweight solution that combines the speed of direct memory access with efficient LZ4 compression. It reduces write latency significantly compared to standard compressed formats while maintaining smaller file sizes than uncompressed formats.

工作流

• Install the numpy-cache library.
• Import the save and load functions from numpy_cache.
• Call save(array, filename) to compress and write the array using LZ4.
• Optionally adjust the acceleration parameter (1-16) to trade compression ratio for speed.
• Call load(filename) to decompress and restore the array with original dtype and shape.

证据

numpy_cache: ~100 ms for 100 MB array

来源

20 × faster than np.savez_compressed()

来源