[Python] Pandas ThreadPoolExecutor with 16 workers cause missing or None DataFrames, while...

I'm working on a real-world data processing pipeline where I process between 300–500 Excel files daily. Each file is read, wrangled into a pandas.DataFrame, and at the end all the resulting DataFrames are concatenated into one.

When I use 1 worker (max_workers=1), everything works 100% fine, every time.

But when I switch to multi-threading with 16 workers (ThreadPoolExecutor(max_workers=16)), I encounter major issues:

Only about 20–50 files (out of ~300) get successfully processed and concatenated.

The rest result in None, or sometimes an error like: TypeError: cannot concatenate object of type 'NoneType'

Some threads produce empty DataFrames even though the source files are valid.

I am not modifying any shared DataFrame between threads. Each thread processes its own file and returns a new DataFrame.

To illustrate this issue, I've written a simplified, version

that mimics the structure of my real function (the real one is more complex, involving more wrangling and additional columns)

import pandas as pd
from concurrent.futures import ThreadPoolExecutor, as_completed

def process_file(file_path):
try:
df = pd.read_excel(file_path)
df['date'] = pd.to_datetime(df['date'], errors='coerce')
df['foo'] = df['foo'].astype(str).str.upper()
if df.empty:
print(f"[WARN] Empty DataFrame from: {file_path}")
return df
except Exception as e:
print(f"[ERROR] Failed to process {file_path}: {e}")
return None

def process_files_in_parallel(file_list, max_workers=16):
results = []
failed_files = []

with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(process_file, file): file for file in file_list}

for future in as_completed(futures):
file = futures[future]
try:
df = future.result()
if df is not None and not df.empty:
results.append(df)
else:
failed_files.append(file)
except Exception as e:
print(f"[EXCEPTION] {file}: {e}")
failed_files.append(file)

print(f"[SUMMARY] Successfully processed: {len(results)} files")
print(f"[SUMMARY] Failed or empty: {len(failed_files)} files")

if not results:
raise ValueError("No valid dataframes to concatenate.")

final_df = pd.concat(results, ignore_index=True)
return final_df


Example Error Output (Real Case):

[WARN] Empty DataFrame from: file_24.xlsx
[ERROR] Failed to process file_35.xlsx: [Errno 2] No such file or directory
[EXCEPTION] file_58.xlsx: cannot concatenate object of type '<class 'NoneType'>'
...
[SUMMARY] Successfully processed: 23 files
[SUMMARY] Failed or empty: 277 files


What I've confirmed:

Each thread works on completely independent files. No shared state.

All files exist and are valid Excel files.

When running the same logic with max_workers=1, it consistently succeeds.

The issue arises only when increasing the number of workers.

My guess is this might be due to:

Thread-safety issues with pandas.read_excel or the underlying engine?

OS-level I/O limits?

Too many concurrent reads on disk?


Question:

Why does ThreadPoolExecutor with multiple workers (e.g., 16) cause so many files to return None or empty DataFrames? How can I safely parallelize the reading and wrangling of hundreds of Excel files?

Any insights into best practices for this kind of batch Excel processing with concurrency are appreciated!

Continue reading...