[Python] Update the column value if the Id in the dataframe is in the list

I am working on a very large dataset that has over 800,000 records and I'm trying to update the column value based on the ID column. If

# Imports
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName('SparkByExamples.com') \
.getOrCreate()

data = [('James','Smith','M','N'), ('Anna','Rose','F','N'),
('Robert','Williams','M','N')
]
columns = ["firstname","lastname","gender","deleted"]
df = spark.createDataFrame(data=data, schema = columns)
df.show()

# Output
+---------+--------+------+-------+
|firstname|lastname|gender|deleted|
+---------+--------+------+-------+
| James| Smith| M| N|
| Anna| Rose| F| N|
| Robert|Williams| M| N|
+---------+--------+------+-------+


I also have a list of names for which I want to update the value of the deleted column.

deleted_list = ['James', 'Robert']


Currently, I'm using a code like this:

df = df.withColumn("deleted_new", when(col("firstname").isin(deleted_list ), "Y").otherwise("N"))
df = df.drop('deleted').withColumnRenamed('deleted_new', 'deleted')


However, it is taking a really long time, as my data frame and list are huge. Is there a PySpark way to make this faster?

# Explected output
+---------+--------+------+-------+
|firstname|lastname|gender|deleted|
+---------+--------+------+-------+
| James| Smith| M| Y|
| Anna| Rose| F| N|
| Robert|Williams| M| Y|
+---------+--------+------+-------+

Continue reading...