I am working on a large Pandas DataFrame which needs to be converted into dictionaries before being processed by another API. The required dictionaries can be generated by calling the .to_dict(orient='records') method. As stated in the docs, the returned value depends on the orient option: Returns: dict, list or collections.abc.Mapping Return a collections.abc.Mapping object representing the DataFrame. The resulting transformation depends on the orient parameter. For my case, passing orient='records', a list of dictionaries is returned. When dealing with lists, the complete memory required to store the list items, is reserved/allocated. As my dataframe can get rather large, this might lead to memory issues especially as the code might be executed on lower spec target systems. I could certainly circumvent this issue by processing the dataframe chunk-wise and generate the list of dictionaries for each chunk which is then passed to the API. Furthermore, calling iter(df.to_dict(orient='records')) would return the desired generator, but would not reduce the required memory footprint as the list is created intermediately. Is there a way to directly return a generator expression from df.to_dict(orient='records') instead of a list in order to reduce the memory footprint? Continue reading...
