[Python] What is the most optimal choice and best practice when iterating through datasets

So im trying to iterate through some fundus images but my ipynb file keeps crashing. There are only 3662 images, is there a more optimal way of iterating through my entire data? The sole purpose for this is just visualization and exploring the data.Therefore i will use a dataloader once the data is cleaned but for now i'd like to iterate over the entire dataset

#load in the train data - explore and clean
training_images=[]
missing_labels=[]
#count the sample distribution per class
count={"No DR" :0 , "Mild DR": 0, "Moderate DR": 0, "Severe DR": 0, "Proliferative DR" : 0}



#iterate through the data in subsets, iterating through 200 images at a time
for chunk in np.array_split(df, 1000):
for idx ,i in tqdm(chunk.iterrows(), total=len(chunk)):
try:
img_path = os.path.join(path, f"{i['id_code']}.png")
img = cv2.imread(img_path, cv2.IMREAD_COLOR)
label = i["diagnosis"]

if pd.isna(label):
#if any image within the csv file does not contain a label, we append the image to the missing_labels, in which we can later filter out
missing_labels.append(np.array(img))

#convert each image into a one hot encoding
#np.eye() allows you define the number of output classes and the [] after it defines which class will be hot
#np.eye(2)[0] for instance meaning [1,0], we have 2 classes and the 0th index is hot meaning that is our true label
training_images.append([np.array(img),np.eye(5)[int(i["diagnosis"])]])

except Exception as e:
print(f"There was an error processing Image : {img_path}")

print(training_images[0][0].shape)
print(training_images[0])

#as the label and image will be a list of lists, this should return a numpy array and its corresponding one hot encoded
print(f"The number of msising labels found in the dataset are : {len(missing_labels)}")

Continue reading...