[Python] How to package a PySpark + Delta Lake script into an EXE with PyInstaller

I’m trying to convert my PySpark script into an executable(.exe) file using PyInstaller. The script runs fine in Python, but after converting to an EXE and executing it, I get the following error:

'JavaPackage' object is not callable
Traceback (most recent call last):
File "job8.py", line 34, in <module>
.getOrCreate()
File "pyspark\sql\session.py", line 559, in getOrCreate
File "pyspark\sql\session.py", line 635, in __init__
TypeError: 'JavaPackage' object is not callable


Here is my PySpark code snippet:

import logging
import traceback
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
import re
import time
from pyspark.sql.functions import to_date, col, to_timestamp, trim

log_file = "C:\\BI\\Data\\gdelta_job.log"
logging.basicConfig(
filename=log_file,
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)

try:
spark = SparkSession \
.builder \
.appName("Delta with PySpark") \
.config('spark.jars.packages', 'io.delta:delta-spark_2.12:3.1.0') \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("spark.sql.legacy.timeParserPolicy", "LEGACY") \
.config("spark.driver.memory", "4g") \
.config("spark.sql.warehouse.dir", "C:\\BI\\Data\\") \
.master("local[*]") \
.enableHiveSupport() \
.getOrCreate()

except Exception as e:
print(e)
logging.error(f"Spark job failed: {e}")


And here’s the PyInstaller command I used to convert the script:

pyinstaller --onefile Job8.py ^
--add-data "C:\BI\Program Files\Python\Python310\Lib\site-packages\pyspark\errors;pyspark\errors"


Could you please help to resolve error and convert into exe file

Continue reading...