I'm new to Python (I'm used to C++ programming), and I've written a script (using C++-based style) that gets sets of five parameters from a number of files and stores in lists to process them afterwards. I've observed that the script runs slower and slower as it process every parameter file. This is the code: for i in range (1, (nbr_files + 1)): param1_list = [] param2_list = [] param3_list = [] param4_list = [] param5_list = [] # Open the file with the parameter list param_file = "./param_file_no_{0:0>9d}.txt".format(i) f = open(param_file, 'r') for line in f: [param1, param2_txt, param3, param4_txt, param5_txt] = line.strip().split(",") # Convert numeric parameters to numbers param2 = int(param2_txt) param4 = int(param4_txt, 16) param5 = int(param5_txt) param1_list.append(param1) param2_list.append(param2) param3_list.append(param3) param4_list.append(param4) param5_list.append(param5) # Close the file with the parameter list f.close() # Process parameter lists ... Note that the number of files ("nbr_files") is known previously (and can be very high, up to one million), and the number of sets of parameters in each file is variable. I've realised that each iteration of the "for i..." loop runs slower and slower, but I don't understand why. Perhaps Python does not manage memory in every loop iteration as I expected? I assumed that the lists created in every iteration of "for i..." loop would be destroyed after ending the iteration, but maybe I'm wrong? Should I clear the lists on my own after every loop iteration, instead? What am I doing wrong? How could I improve the memory usage and fix the progressive slowdown of the script? I've tried to find similar questions in the chat, but I haven't found any. Just in case, what I do to "Process parameter lists" is to simulate a Pytorch CNN. I don't think that running a CNN a lot of times should influence in the duration of the simulation, should it? Thanks! Continue reading...
