I'm writing a script that process texts contained in JSON objects, saved in a ndjson file. The aim is to filter those texts and make them available for analysis. The file is 3.5 GB, with almost 7 millions rows in JSON format. During this process I need to cycle over the list of the JSONs and check if some conditions are satisfied. My issue is that the process in really slow and I need to make it faster. The structure of the for cycle is as follows: for json_obj in json_list: some_scipy_function if condition_is_satisfied: json['new_key'] = new_value with open('file_name.ndjson', 'a') as file: file.write(json.dumps(json_obj)+'\n') Is there a way to optimize for cycles in Python and let it use more CPU and RAM? At the moment, it uses only 3.5 GB out of 64 GB of RAM and only 8.5% of CPU. Continue reading...
