I'm trying to figure out how to aggregate rows into batches (ie. a sliding / tumbling window) such that if we had a table like id x y ---- ---- ---- 1 x1 y1 2 x2 y2 3 x3 y3 4 x4 y4 5 x5 y5 The result would be id x y Grouped x3 ---- ---- ---- ----------------------------------------- 1 x1 y1 None 2 x2 y2 None 3 x3 y3 [[1, x1, y1], [2, x2, y2], [3, x3, y3]] 4 x4 y4 [[2, x2, y2], [3, x3, y3], [4, x4, y4]] 5 x5 y5 [[3, x3, y3], [4, x4, y4], [5, x5, y5]] I currently have an aggregated column by using concat_list such that: id x y List ---- ---- ---- ------------- 1 x1 y1 [1, x1, y1] 2 x2 y2 [2, x2, y2] 3 x3 y3 [3, x3, y3] 4 x4 y4 [4, x4, y4] 5 x5 y5 [5, x5, y5] I just don't know how to aggregate them across rows like the example above. Thanks so much! Continue reading...
