[Python] How to replace a value including the column in a structure

When I use DataFrame.replace, it doesn't replace the values that are in a structure.

In this example, it doesn't replace the value of my_struct.struct_string:

from awsglue.context import GlueContext
from pyspark.context import SparkContext
from pyspark.sql.functions import col
from pyspark.sql.types import StringType, StructType, StructField

glueContext = GlueContext(SparkContext.getOrCreate())

data = [
("null", {"struct_string": "null"}),
]

schema = StructType([
StructField("a_string", StringType(), True),
StructField(
"my_struct",
StructType([
StructField("struct_string", StringType(), True),
]),
True
)
])


df = spark.createDataFrame(data, schema)


df = df.replace("null", None)

df_astring = df.filter(col("a_string").isNotNull())
df_struct_string = df.filter(col("my_struct.struct_string").isNotNull())

print("My df_astring")
df_astring.show()
print("My df_struct_string")
df_struct_string.show()


It print:

My df_astring
+--------+---------+
|a_string|my_struct|
+--------+---------+
+--------+---------+

My df_struct_string
+--------+---------+
|a_string|my_struct|
+--------+---------+
| null| {null}|
+--------+---------+


Note that I also tried df = df.replace("null", None, ["a_string", "my_struct.struct_string"]), but I get the exception java.lang.UnsupportedOperationException: Nested field my_struct.struct_string is not supported.

The solution needs to be dynamic, so it won't precise manually the column name that are string.

Continue reading...