Why is os.scandir() significantly faster than os.walk() in Python?

Feedback from a simple benchmark in a real-world environment

When developing synchronization, indexing or file-processing tools (backup, directory scanning, etc.), recursive traversal of the file system quickly becomes a critical performance point.

In our tests, we wanted to concretely compare two common approaches in Python:

• os.walk() (historical method, widely used)
• os.scandir() (more recent, lower-level API)

The goal: a simple benchmark, without artificial optimization, run on real folders, with a few hundred to several tens of thousands of files.

The benchmark

The test consists of:

• recursively traversing a folder
• counting the total number of files
• measuring the execution time

Same logic, same machine, same business code.
The only difference: the traversal method.

Observed results

Here are some representative excerpts from the tests carried out:

👉 In every case, os.scandir() is about 2× faster.

And the larger the tree, the more significant the gap becomes.

Why this difference?

1. The hidden cost of stat()

os.walk() relies on a higher-level abstraction.
When it traverses files, Python makes many stat() system calls to determine whether an entry is a file or a folder.

These calls are:

• costly on disk
• even slower on network shares (SMB, NFS, NAS)
• multiplied by the total number of entries

2. os.scandir() works at a lower level

os.scandir() directly uses the information returned by the file system (via dirent on Linux).

Result:

• the type (file/folder) is often already known
• no systematic stat() needed
• far fewer kernel ↔ userland transitions

This is exactly what explains the massive performance gain.

“But doesn’t os.walk() already use scandir?”

Yes… internally, since Python 3.5.

But:

• os.walk() remains a general-purpose API
• it generates full tuples (root, dirs, files)
• it does more work than necessary in simple cases

👉 When you simply want to traverse quickly, using os.scandir() directly remains unbeatable.

Why choose Python, despite lower performance than some languages?

It is widely accepted that Python is, on a purely technical level, slower than compiled languages like C, C++ or Rust. This reality is not, however, a flaw in itself, but rather a deliberate trade-off. Choosing a language is never limited to raw performance alone: it involves considerations of budget, complexity, maintainability and long-term technical debt.

Python stands out above all for its simplicity of design and readability. Clearer code reduces development time, makes code reviews easier, limits errors and allows teams to ramp up faster. For the same budget, this translates into more features delivered, faster, and with a significantly lower project risk.

Moreover, not all projects require maximum optimization from the outset. In many cases, performance is not the main limiting factor, or can be improved locally, only where it is needed. Python precisely allows this progressive approach: start simple, then optimize in a targeted way, without premature over-engineering.

Finally, technical debt is often underestimated when more complex technologies are favored from the start. Very high-performing languages that demand more expertise can generate high costs in maintenance, recruitment and functional evolution. Python, by reducing this initial complexity, offers a lasting balance between efficiency, scalability and cost control.

In short, a bit of love for Python 😉

Critical use cases where the difference matters

In our projects, this kind of gain is far from theoretical. It is decisive for:

• 🔄 synchronization tools
• 📂 standalone folder scanners with several tens of thousands of files
• 🧾 data processing pipelines
• 🌐 file servers
• 🧠 indexing engines

Halving scan time means:

• less CPU load
• fewer disk I/O operations
• less perceived latency
• and better overall scalability

Recommendation

👉 For any new Python development involving recursive file traversal:

• prefer os.scandir() with explicit recursion
• reserve os.walk() for simple or non-critical scripts

This choice alone can turn an “acceptable” script into a truly industrial tool.

If you work on performance, storage, automation or large-scale file processing issues, I can support you from architecture all the way to low-level optimization.

📩 Got a project? Get in touch: Arthur Perrot, infrastructure & cybersecurity expert.