[Python] How to efficiently perform parallel file search using pathlib `glob` in Python for...

I'm working on a Python project where I need to search for specific files across a very large directory structure. Currently, I'm using the glob (or rglob) method from the pathlib module, but it is quite slow due to the extensive number of files and directories.

Here's a simplified version of my current code:

from pathlib import Path

base_dir = Path("/path/to/large/directory")
files = list(base_dir.rglob("ind_stat.zpkl"))


This works, but it's too slow because it has to traverse through a massive number of directories and files. Ideally, I'd like to divide the directory traversal work across multiple threads or processes to improve performance. Are there optimizations or alternative libraries/methods that could help improve the performance?

Continue reading...