[Python] Getting incorrect order of files using for loop when referencing a CSV file of the...

I have a CSV file that reads like this:

Week File Name Week File Name
1 results_tdr_periods_4_1 1 results_tdr_periods_8_1
4 results_tdr_periods_12_1 4 results_tdr_periods_16_1
8 results_tdr_periods_20_1 8 results_tdr_periods_24_1
12 results_tdr_periods_32_1 12 results_tdr_periods_28_1
16 results_tdr_periods_36_1 16 results_tdr_periods_40_1
20 results_tdr_periods_48_1 20 results_tdr_periods_44_1
24 results_tdr_periods_52_1 24 results_tdr_periods_56_1

And so on and so forth.

I have the associated data files in a folder. The folder path goes All Tests -> results_tdr_periods_[given_number]_1 -> [the file I need].

Therefore, I'm using the code below to pull from those folders using the CSV file above. I want to compare week 1 against week 1, week 4 against week 4, etc. (I am using Google Colab, which is why my code is like this.)

!pip install patool
import patoolib
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
import pandas as pd
import os
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
import itertools

patoolib.extract_archive("All Tests.zip",outdir="/content")
#I have my All Tests folder in a zip in order upload it to colab

cvar_list2 = []
subtraction = []
list1 = np.arange(14)
label = []
xticks_label = []

file_list = pd.read_csv("[my csv file above]")
#I have my own csv file, but i'm just labeling it like this to show I'm using the CSV data from above

def get_costdata(dataframe1,dataframe2,clist1,clist2,row_name):
cvar1 = dataframe1.loc[dataframe1['Costs'] == row_name]

cvar1_z1 = f'{float(cvar1["Zone1"]):.3f}'
cvar1_z2 = f'{float(cvar1["Zone2"]):.3f}'
cvar1_z3 = f'{float(cvar1["Zone3"]):.3f}'

clist1.append(float(cvar1_z1))
clist1.append(float(cvar1_z2))
clist1.append(float(cvar1_z3))
# print(clist1)

#second round info
cvar2 = dataframe2.loc[dataframe2['Costs'] == row_name]

cvar2_z1 = f'{float(cvar2["Zone1"]):.3f}'
cvar2_z2 = f'{float(cvar2["Zone2"]):.3f}'
cvar2_z3 = f'{float(cvar2["Zone3"]):.3f}'

clist2.append(float(cvar2_z1))
clist2.append(float(cvar2_z2))
clist2.append(float(cvar2_z3))
# print(clist2)

rootdir = "/content/All Tests"

for subdir, dirs, files in os.walk(rootdir):
last_folder = os.path.basename(subdir)
for file in files:
if file != "costs.csv":
continue # Skip irrelevant files
for idx, value in enumerate(file_list["File Name"]):
if last_folder == file_list["File Name"][idx]:
#my main issue is here
#trying to do 'if the folder is equivalent to the index's folder in the column "file name", then use the costs.csv file from that folder
print(last_folder)
new_var1 = pd.read_csv(os.path.join(subdir, file))
if last_folder == file_list["File Name.1"][idx]:
#same thing as above but with the second column of file names instead
print(last_folder)
new_var2 = pd.read_csv(os.path.join(subdir, file))

# Only run if new_cvar1 is already defined
if 'new_var1' in locals():
get_costdata(new_var1, new_var2, cvar_list1, cvar_list2, "cVar")
label.append("Week: " + str(file_list["Week"][idx]))
else:
print(f"Skipping idx {idx}: new_cvar1 not found for {last_folder}")


However, when I run my code, using print(last_folder) to see which folder's data file (cost.csv) is being used, it does them completely out of order. I restarted my runtime to double check everything and it gave me a different order of last_folders each time. This is what I have so far:

results_tdr_periods_16_1

results_tdr_periods_52_1

results_tdr_periods_108_1

results_tdr_periods_112_1

results_tdr_periods_44_1

results_tdr_periods_32_1

results_tdr_periods_56_1

results_tdr_periods_84_1

results_tdr_periods_20_1

results_tdr_periods_68_1

results_tdr_periods_88_1

results_tdr_periods_40_1

results_tdr_periods_24_1

results_tdr_periods_104_1

results_tdr_periods_72_1

results_tdr_periods_12_1

results_tdr_periods_28_1

results_tdr_periods_80_1

results_tdr_periods_64_1

results_tdr_periods_8_1

results_tdr_periods_76_1

results_tdr_periods_100_1

results_tdr_periods_48_1

results_tdr_periods_60_1

results_tdr_periods_36_1

results_tdr_periods_96_1

results_tdr_periods_92_1

results_tdr_periods_4_1

As you can see, these are not taking the files from the folders that are supposed to be used, and are not in order. I do not know how to make them go in order. I could be using the print function wrong, but I also double checked the data that's being pulled and it's not the correct data either. I feel like I'm doing something wrong with the order of the if statements or the indentation, but I can't figure out what it is nor how to fix it.

Continue reading...