[Python] How do I fix for the code output and optimize the read csv process?

I am reading a csv (the Moral Machine dataset) that is 10GB. The following code tries to reshape the data by pairing rows together with the same "ResponseID" using Dask. I am only using one ID for a test, but it's already using so much memory as it tries to fetch 2 rows.

Even when it does run, it returns not only one row that I needed but also other rows that I don't need...I don't know why

I need to optimize the code so it runs faster and I want two rows that I needed.

import numpy as np
import dask.dataframe as dd

# Assuming 'ddf' is a Dask DataFrame already created from a large dataset
# Process data in chunks to avoid memory issues
chunk_size = 5 # Adjusted chunk size to reduce memory usage

# First id for test
for id in collection[:1]:
location = np.where(responseID == id)
print(location)

# Check if there are enough locations to avoid IndexError
if len(location[0]) > 1:
# Use Dask's .iloc to access rows without computing them immediately
new_row1 = ddf.loc[location[0][0]].to_dask_array() # Access first row, all columns
new_row2 = ddf.loc[location[0][1]].to_dask_array() # Access second row, all columns

# Compute only when necessary
print(new_row1.compute()) # Compute first row
print(new_row2.compute()) # Compute second row
else:
print("Not enough locations found for id:", id)


I have tried to use other functions, converting it to array (which prevents memory errors), checked the logic with what I knew...

I was expecting the 2 rows returned by print statement but it instead returns a giant array full of the 2 rows I need and many other rows.

Continue reading...