I have a Pyspark dataframe that I want to perform batch processing on. The DF have an id and a count. | ------ | ----- | | ID | Count | | ------ | ----- | | abc | 500 | | def | 300 | | ghi | 400 | | jkl | 200 | | mno | 1100 | | pqr | 900 | I want to create batches (list of IDs) that sum up to the first time that it exceeds a threshold. Lets say 1000. In above the first batch would be ['abc','def','ghi'] (500+300+400)=1200. The second batch would be ['jkl','pqr'] (200+900)=1100. ['mno'] should be a batch by itself. I already developed the code to strip out the 'mno' as the first step. But how can I develop an iterative process to create the rest of the batches? The order of the IDs do not matter. If I can avoid the processing of an ordered DF first I would prefer that. Continue reading...
