[Python] <class 'TypeError'>: Object of type 'DataFrame' is not JSON serializable

I have a dataframe with 3 columns : date, week and yearweek. I want to extract the first and the last day of the week. Here is my dataset :

+----------+-------+------------+
| date| week| yearweek|
+----------+-------+------------+
|2025-01-26| 5| 2025S05|
|2025-01-27| 5| 2025S05|
|2025-01-28| 5| 2025S05|
|2025-01-29| 5| 2025S05|
|2025-01-30| 5| 2025S05|
|2025-01-31| 5| 2025S05|
|2025-02-01| 5| 2025S05|
+----------+-------+------------+


I need this result, that's why I need a list :

['2025-01-26',
'2025-01-27',
'2025-01-28',
'2025-01-29',
'2025-01-30',
'2025-01-31',
'2025-02-01']


I tried to do this :

list_date_week = list(
df.select(col('date')).toPandas()['date']
)
min_date=min(list_date_week)
max_date=max(list_date_week)


But I have this error :

<class 'TypeError'>: Object of type 'DataFrame' is not JSON serializable​

I tried another solution to convert my Pyspark column to List.

When I try this 2 codes I have this error :

list_date_week = df.select("date").rdd.flatMap(lambda x: x).collect()

list_date_week = df.rdd.map(lambda x: x[0]).collect()

AttributeError: 'SparkSession' object has no attribute 'serializer'
​

df is a dataframe and list_date_week is a list. I'm beginner in Pyspark, how can I convert my dataframe to list without error in Pyspark ?

Continue reading...