[Python] How to apply the describe function after grouping a PySpark DataFrame?

I want to find the cleanest way to apply the describe function to a grouped DataFrame (this question can also grow to apply any DF function to a grouped DF)

I tested grouped aggregate pandas UDF with no luck. There's always a way of doing it by passing each statistics inside the agg function but that's not the proper way.

If we have a sample dataframe:

df = spark.createDataFrame(
[(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
("id", "v"))


The idea would be to do something similar to Pandas:

df.groupby("id").describe()


where the result would be:

v
count mean std min 25% 50% 75% max
id
1 2.0 1.5 0.707107 1.0 1.25 1.5 1.75 2.0
2 3.0 6.0 3.605551 3.0 4.00 5.0 7.50 10.0


Thanks.

Continue reading...