[Python] How to check pyspark dataframe column for incorrect value type using pytest?

I am trying to write a test to see if the spark dataframe has records with incorrect value type, but I'm stuck. There is the dataframe:

schema1 = StructType(
[
StructField("id_key", IntegerType(), True),
StructField("country", StringType(), True),
]
)
data_wrong = [(10, "France"), (20, "USA"), (30, 1790), (40, "Germany")]

df = spark_session.createDataFrame(data=data_wrong, schema=schema1)


And now I want to catch that there is a wrong datatype in the country field in the 3rd record.

I tried using

• assert isinstance(df["country"], str)
• assert isinstance(df.select("country"), str)
• Looping through the columns and checking their types:

for c in df.columns:
if df[c].dtype != 'string':
print("Error")


of course none of them work and I'm out of ideas. Please help

Continue reading...