[Python] Some CSV outputs are empty in parallel Python processing, even though the code path...

I have a Python script that processes a batch of CSV files in parallel. Each input CSV has exactly 200 rows, so I use an if num_videos == 200: check to run a function like:

if num_videos == 200:
all_data = process_videos_in_batches(video_ids, batch_size, API_KEY)
output_csv_path = os.path.join(output_dir, os.path.basename(file_path))
save_all_data_to_csv(all_data, output_csv_path)


Inside process_videos_in_batches, I add debug lines:

print(f"Debug: {len(all_data)}")
print(f"Debug: {all_data[:1]}")


And inside save_all_data_to_csv:

final_df.to_csv(output_csv_path, index=False, encoding='utf-8')
print("Debug: writing CSV to", output_csv_path)


For some of the output CSV files, I see the debug prints (and the file contains many rows, as expected). However, for other files, the CSV is created but ends up empty. Even more strangely, the debug prints for these “empty” files never appear—so it looks like the code isn’t running that section fully, yet a CSV still appears. Furthermore, it seems unlikely that the process inside process_videos_in_batches would result in a CSV with 0 rows, but in reality, some output CSV files have 0 rows (the full code of the main function and parts of the outputs are provided at the end). I have confirmed that files where the lines

print(f"Debug: {len(all_data)}")
print(f"Debug: {all_data[:1]}")


are correctly printed have content, whereas files where these lines are not printed are empty.

• I’ve confirmed each input CSV really has 200 rows (so the if check should pass)
• I’m using unique output filenames, so it’s not a file-overwrite issue.
• I checked htop and the system isn’t low on memory or CPU.

My environment is:

• Python: 3.10.15
• Conda: 24.9.2
• VSCode: 1.96
• OS: Ubuntu 22.04.5 LTS

Why might some files run the full code path (with debug prints and data in the CSV) while others apparently skip the debug prints and produce an empty CSV—despite the if condition being satisfied for all files? Could something in my parallel execution be causing the function to exit early or fail silently for certain files, even though the CSV is still created?

Any tips or insights would be greatly appreciated. Thanks in advance!

--- the full code of main function ---

process_videos_in_batches

def process_videos_in_batches(video_id_list, batch_size, API_KEY):

all_data = []

all_video_ids = video_id_list[:]


while all_video_ids:
batch_video_ids = all_video_ids[:batch_size]
all_video_ids = all_video_ids[batch_size:]

videos_info = get_video_data(batch_video_ids, API_KEY)

### Debug ###
print(f'Debug: {videos_info}, {batch_video_ids}')
#############

# videos_infoがNoneの場合は次のバッチへ
if videos_info is None:
continue

for video_id in batch_video_ids:
video_data = videos_info.get(video_id) if videos_info else None
if video_data:
comments = get_all_video_comments(video_id, API_KEY)
for comment in comments:
# 親コメントの情報を展開
video_entry = video_data.copy()
video_entry.update({
'comment': comment.get('comment', pd.NA),
'comment_like_count': comment.get('comment_like_count', pd.NA),
'comment_author_channel_id': comment.get('comment_author_channel_id', pd.NA),
'comment_published_at': comment.get('comment_published_at', pd.NA),
'updated_at': comment.get('updated_at', pd.NA),
'has_reply': comment.get('has_reply', 'no'),
'reply_comment': comment.get('reply_comment', pd.NA),
'reply_like_count': comment.get('reply_like_count', pd.NA),
'reply_author_channel_id': comment.get('reply_author_channel_id', pd.NA),
'reply_published_at': comment.get('reply_published_at', pd.NA),
'reply_updated_at': comment.get('reply_updated_at', pd.NA)
})
all_data.append(video_entry)
### Debug ###
print(f'Debug: {len(all_data)}')
print(f'Debug: {all_data[:1]}')
#############
return all_data


save_all_data_to_csv

def save_all_data_to_csv(all_data, output_csv_path):
# DataFrame に変換
final_df = pd.DataFrame(all_data)

# CSV に保存
final_df.to_csv(output_csv_path, index=False, encoding='utf-8')
### Debug ###
print("Debug: writing CSV to", output_csv_path)
#############


process_single_file

def process_single_file(file_path, output_dir, batch_size, API_KEY):
"""
1つのCSVファイルを処理し、必要に応じて結果をCSVで出力する。
"""
df = pd.read_csv(file_path, encoding='utf-8')
video_ids = list(df['related_video_id'])
num_video = len(video_ids)
if num_video == 200:
all_data = process_videos_in_batches(video_ids, batch_size, API_KEY)
output_csv_path = os.path.join(output_dir, os.path.basename(file_path))
save_all_data_to_csv(all_data, output_csv_path)

# 終わったらファイルパスを返す（ログ代わり）
return file_path


The main process:

input_files = os.listdir(root_dir)
file_paths = [os.path.join(root_dir, f) for f in input_files if os.path.isfile(os.path.join(root_dir, f))]

# 並列実行する
num_workers = 4

futures = []
with ProcessPoolExecutor(max_workers=num_workers) as executor:
for file_path in file_paths:
# process_single_file関数を並列に実行
future = executor.submit(process_single_file, file_path, output_dir, batch_size, API_KEY)
futures.append(future)

# 処理が終わったら順に結果を取得
# tqdmを使って「終わった数」を確認するときは、as_completedに対して進捗バーを回します
for f in tqdm(as_completed(futures)):
try:
finished_file = f.result()
except Exception as e:
print("Error happened in worker:", e)


Outputs:

Debug: {'lAtasG8EVEg': {'video_id': 'lAtasG8EVEg'...
...
Debug: 22058
Debug: [{'video_id': '8BtA6fO93_w'...
Debug: writing CSV to /home/foo/mnt/vt/related_videos/data_info/C_dsxOR9JJw.csv

Continue reading...