FlexReadDir: up to 3.8× faster directory listing in Go

Description and use

FlexReadDir speeds up directory listing in Go up to 3.8× faster by eliminating per-file system calls and reducing allocations.

Problem

Directory listing in Go is slow because it performs an extra system call per file on macOS and Linux, leading to many allocations and high latency for tools that scan directories frequently.

Outcome

On macOS, FlexReadDir is 3.8× faster than os.ReadDir+Info for 1,000 files; on Linux 1.6× faster; on Windows 1.3× faster, as shown by median benchmark results.

Workflow

• Install the library with go get github.com/pbednarcik/flexreaddir/frd
• Call frd.ReadDir(".") to obtain a sorted slice of Entry structs
• Iterate over entries, using e.Name(), e.Size(), e.ModTime() for processing
• Compare current entries with previous scan using == on Entry structs to detect changes

Evidence

On macOS and Linux, that's one call to list the directory, then one lstat for every file.

Source

FlexReadDir here is AppendDir, unsorted and reusing its result slice, as a walker would use it.

Source

FlexReadDir is 3.8× faster on macOS, 1.6× on Linux, 1.3× on Windows for 1,000 files.

Source