[Python] pyspark some task failed but retry successfully, however, the responding parquet file...

I run a pyspark job, the job stopped sucessfully, but when i check the result, i found that partial data seemed missing. then i check the log, here is some thing i found:

1. error inside log file

TaskSetManager: Lost task 7.0 in stage 5.0 (TID 549) (worker executor 1): org.apache.spark.SparkException: Task failed while writing rows.
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:500)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:324)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$16(FileFormatWriter.scala:229)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.scheduler.Task.run(Task.scala:131)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at java.lang.Thread.run(Thread.java:748)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - Caused by: java.io.FileNotFoundException: xxxxx/_temporary/0/_temporary/attempt_202411281735309015966430992825558_0005_m_000007_549/dt=2024-11-19/region=br/part-00007-996f4f11-c9b2-4552-8b56-c6dfbba5c4b8.c000.snappy.parquet: not found
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at io.juicefs.JuiceFileSystemImpl.error(JuiceFileSystemImpl.java:196)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at io.juicefs.JuiceFileSystemImpl.access$500(JuiceFileSystemImpl.java:73)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at io.juicefs.JuiceFileSystemImpl$FSOutputStream.close(JuiceFileSystemImpl.java:1002)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:1106)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:132)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:164)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:75)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseCurrentWriter(FileFormatDataWriter.scala:65)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:87)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:117)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:308)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:314)
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - ... 9 more
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] -
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] - 24/11/28 17:35:34 INFO [dispatcher-CoarseGrainedScheduler] TaskSetManager: Starting task 7.1 in stage 5.0 (TID 552) (worker-4, executor 8, partition 7, PROCESS_LOCAL, 4466 bytes) taskResourceAssignments Map()
[2024-11-28, 17:35:34 CST] {subprocess.py:93} INFO - 2024-11-28 17:35:34 - INFO - root - __init__.py:__execute_command:197 - [28755] -

1. then i review the job execution detail on spakr web gui. from the above log we can know that #549 task failed at the first time, then spark restart #552 task the recompute and it run successfuly.
2. however, when i checked the path of result table, i did find the one parquet file is missing

Finally I try to run the my code again, and this time the result is correct.

According to my observation, this problem happens intermittently.

python version: 3.6 spark version: 3.2.0

This borders me because i am not able to know if the job actually run successfully or not

Continue reading...