[Python] PySpark - Applying User defined function (UDF) to entire row of dataframe

I am quite new to Pyspark, though frequent user of pandas etc.

I have a spark dataframe consisting of columns: ID, Trajectory, type

Where ID is an integer, Trajectory is a list of dataframe-rows containing coordinate-information and type is a list of strings

The goal is to, for each ID, draw a polyline in a python folium-map, based on the coordinates in the Trajectory-column-list.

as reference:

df.first() produces the following outcome

Row(ID=209223000, Trajectory=[Row(Timestamp=datetime.datetime(2024, 11, 1, 2, 58, 29), Latitude=55.108018, Longitude=18.306343, Destination='Unknown'), Row(Timestamp=datetime.datetime(2024, 11, 1, 6, 18, 18), Latitude=55.115625, Longitude=18.103083, Destination='Unknown')], types=['Undefined'])


So to be clear, I have no need for an extra column being added to the dataframe, and no objections if that is neede either.

So the Pseudo would look something lke this:

create map-object m

For each Row in df:
create polyline object ply
ply.coordinates = [item.Lat,item.Long] for item in Row['Trajectory']
ply.tooltip = Row['ID'] + Row[types]
add ply to m


The end goal is to showcase the tracks of objects depending on certain filters being applied in the dataframe

I've tried to create an UDF with the pseudocode described, but I cant wrap my head around how to apply the UDF, since a UDF is applied on a column rather than a dataframe.

The expected result would be add polylines to the map-object based on each row of the dataframe, e.g. each individual ID.

Continue reading...