[Python] PySpark coalesce(1) is slow when combining many small JSON files into one — faster...

I'm using PySpark on Databricks to process JSON files stored in Google Cloud Storage. Each file in the input directory contains a single JSON object, and I want to combine all of them into a single JSON file and write it back to the same directory.

Here’s a simplified version of my code:

import json
from datetime import datetime, timedelta

BUCKET = "bucket_name"
data_types = ["1", "2", "3"]
BUCKET_PATH = f"gs://{BUCKET}"

dbutils.widgets.text("date", "ERROR")
date_str = dbutils.widgets.get("date")
date_format = '%Y-%m-%d'
yesterday = (datetime.strptime(date_str, date_format) - timedelta(1)).strftime(date_format)

def process_joined_json_file(bucket_path: str, data_type: str, date: str, mode: str = "overwrite"):
input_path = f"{bucket_path}/{data_type}/{date}"
output_path = input_path

df = spark.read.format("json").load(input_path)
df.coalesce(1).write.mode(mode).json(output_path)

for data_type in data_types:
process_joined_json_file(BUCKET_PATH, data_type, yesterday)


The issue is that this runs very slowly, even though the total number of input files is less than 1 million and each file contains just one small JSON object.

I suspect the bottleneck is coalesce(1), which forces a single partition and a shuffle.

Is there a faster way to combine many small JSON files into one in PySpark? I’m okay with the output being a single JSON file (array or newline-delimited), as long as performance improves.

Continue reading...