[Python] Number of completed tasks exceeds total tasks in PySpark

I have the following (simplified) code running in PySpark.

df is a PySpark dataframe with 2B rows. I keep hitting resource issues when running such a large dataframe, so I'm trying to get a subset of the dataframe to work on.

df_10M = df.limit(10000000)
df_a = df_10M.withColumn(...).withColumn(...).withColumn(...).withColumn(...)
df_a = df_a.drop("oldColA", "oldColB")
df_a.show(10)


When I run this on Jupyter Notebook, I eventually see

[Stage 4:====================(1572 + 12 / 1122)]

1122 should be the total number of tasks, 1572 should be the number of completed tasks, and 12 is the number of running tasks. If I leave the code running, eventually, I hit the Pod ephemeral local storage usage limit.

My question is how can the number of completed tasks be larger than the total number of tasks?

Continue reading...