[Python] MapType with ArrayType as key throws exception

I can create a DataFrame with a MapType column whose key is an ArrayType. The show() method of the DataFrame works, but the collect() method throws an exception:

> return cloudpickle.loads(obj, encoding=encoding)
E TypeError: unhashable type: 'list'


According to the PySpark documentation, this should be possible: MapType supports keys of type DataType and ArrayType inherits from DataType.

Is this behavior correct?

Code:

from pyspark.sql import SparkSession
from pyspark.sql.types import MapType, ArrayType, StringType, IntegerType

schema = MapType(ArrayType(StringType()), IntegerType())
data = [{("A", "B"): 10, ("X", "Y", "Z"): 20}]
df = spark.createDataFrame(data, schema)
df.show() # works

df.collect() # throws exception


Notes:

1. i want to store sparse arrays in this structure
2. i know that list is mutable and therefore cannot be used as a key in python dicts.
3. i use python 3.11.9 and spark 3.5.3

Continue reading...