[Python] Does converting an RDD to DataFrame and back affect partitions or data structure...

I’m experimenting with Spark and want to clarify how partitioning behaves when converting between RDDs and DataFrames.

Here’s my example:

rdd1 = sc.parallelize([("a",1), ("b",2), ("c",3), ("d",4)], 4)
print(rdd1.glom().collect())
# Output: [[('a', 1)], [('b', 2)], [('c', 3)], [('d', 4)]]

df = rdd1.toDF(["letter", "freq"])
print(df.collect())
# Output: [Row(letter='a', freq=1),
# Row(letter='b', freq=2),
# Row(letter='c', freq=3),
# Row(letter='d', freq=4)]

rdd2 = df.rdd
print(rdd2.glom().collect())
# Output: [[Row(letter='a', freq=1)],
# [Row(letter='b', freq=2)],
# [Row(letter='c', freq=3)],
# [Row(letter='d', freq=4)]]


When converting rdd1 (where each element is a tuple) to a DataFrame df (where each element becomes a Row object), did Spark create new partitions because the data type inside each partition changed from tuple to Row? Or did the partitions remain exactly the same?

Continue reading...