I am trying to make some joints, groupings,... more efficiently with pyspark, by trying to avoid unnecessary exchanges. I have a situation where first I need to join a dataframe by columns (a, b, c), and later another join by columns (a, b, d). I want to bucket by dataframe by the first 2 columns, so that an exchange is not necessary, but spark always force the exchange by the 3 columns before each join. How can I avoid this extra exchanges? There seems to be some PR to get this feature, but they were not merged: https://issues.apache.org/jira/browse/SPARK-18067 https://github.com/apache/spark/pull/29655 Related question: How can I convince spark not to make an exchange when the join key is a super-set of the bucketBy key? At the moment, the only workaround I have seen, that more or less works, is to perform the join with the >= and <= hack. However, this does not serve for groupby operations and joints altogether. It would be ideal to use some kind of hint to tell pyspark that it is safe to join/group with the current bucketing, and that I guarantee that data to be merged/grouped lives within the same node. Continue reading...
