[Python] How to prevent extra backslashes when writing JSONL in Python? [closed]

I'm trying to convert a JSON file into a JSONL file for data preprocessing before training a model. A single line in the JSONL file should contain all JSON objects from the JSON file, where the "text" field is a concatenation of "original" values from the input JSON.

Here's the function I'm using:

import os
import json
import re
from tqdm import tqdm
import logging

logger = logging.getLogger(__name__)

def read_json_dataset(raw_data_folder: str, jsonl_data_folder: str, dataset_name: str):
os.makedirs(jsonl_data_folder, exist_ok=True)
failed_files = []

files = [f for f in os.listdir(raw_data_folder) if f.endswith(".json")]
pattern = re.compile(r'"original"\s*:\s*"(.*?)"', re.DOTALL)

logger.info(f"Reading {dataset_name} dataset")

for filename in tqdm(files, desc="Processing JSON files"):
input_path = os.path.join(raw_data_folder, filename)
output_path = os.path.join(jsonl_data_folder, filename.replace(".json", ".jsonl"))

try:
with open(input_path, "r", encoding="utf-8") as f:
content = f.read()
data = pattern.findall(content)
combined_text = " ".join(data)

with open(output_path, "w", encoding="utf-8") as f_out:
record = {
"text": combined_text,
"metadata": {
"source_file": filename,
"dataset": dataset_name,
"format": "json"
},
}
f_out.write(json.dumps(record) + "\n")

except Exception as e:
logger.warning(f"Error processing {filename}: {e}")
failed_files.append(filename)

logger.info(f"Failed files: {failed_files}")
logger.info(f"Finished reading {dataset_name} dataset")

return failed_files


The problem: In my JSON file, I have entries like this:

[
{
"segmentnr": "BO_K01_D0001_H0001:1b-0",
"original": "@1b #\\/\\/ rgya gar skad du \\/",
"folio": "1b",
"analyzed": "rgya gar skad du "
},
{
"segmentnr": "BO_K01_D0001_H0001:1b-1",
"original": "bi na ya bastu \\/",
"folio": "1b",
"analyzed": "bi na ya bastu "
}
]


The expected output in JSONL should be:

{"text": "@1b #\/\/ rgya gar skad du \/ bi na ya bastu \/"}


However, the actual output is:

{"text": "@1b #\\/\\/ rgya gar skad du \\/ bi na ya bastu \\/"}


Note : I look here Writing special characters to json file generates extra backslashes and notice that backslash is a special character in json. However, these backslashes have meaning in the text, so I must preserve the natural structure.

How can I fix this issue?

Continue reading...