[Python] How to convert PySpark dataframe into a list of dictionaries grouped by a column

I have a PySpark dataframe that looks like:

A B C D E
1 a 0 metric4 value6
1 a 0 metric3 value7
1 b 1 metric2 value5
2 b 0 metric4 value8
2 c 0 metric3 value9

I want to groupby on column A and get following output:

A ListOfDict
1 [{'B':'a', 'C':0, 'D':'metric4', 'E':'value6'}, {'B':'a', 'C':0, 'D':'metric3', 'E':'value7'}, {'B':'b', 'C':1, 'D':'metric2', 'E':'value5'}]
2 [{'B':'b', 'C':0, 'D':'metric4', 'E':'value8'}, {'B':'c', 'C':0, 'D':'metric3', 'E':'value9'}]

Closest I got is with the code

value_cols = [col for col in df.columns if col != "A"]

df_test = df.groupBy("A").agg(F.collect_list(F.struct(*value_cols)).alias("ListOfDict"))


But for this we will get it as a list of arrays but without the column names assigned.

It would be helpful if someone point out issue in above solution or provide an alternate way of doing.

Continue reading...