[Python] Is there a way to optimize concatenation of files instead of appending file data to a...

I'm making a file compiler that takes data from csv files and after every 8 files read concatenates the data into a separate csv file. The program is taking quite long to be able to do these functions. is there a more optimized way to go about this?

I'm currently reading the csv file into a pandas data frame then appending said data frames into a list to compile them for pd.concat() after.

code snippet:

data = \[\]

data_value = pd.read_csv(os.path.join(root, file), sep='\\t') #Reading data into df


data.append(data_value) # appending df to a list

pd.concat(data) #concatenating all data in list into a data frame

Continue reading...