[Python] Kubernetes: driver-py: not found and Missing application resource error

I'm trying to run my pyspark file in a k8s environment using a custom image. It used to work fine with our company provided image but it has spark v2.4.4 so I wanted an upgrade.

First I got this error: /opt/entrypoint.sh: line 128: exec: driver-py: not found

Figured this was due to the driver-py being removed from the entrypoint.sh file in v3+ and so modified my docker image to add the last line.

FROM apache/spark:3.5.1-java17-python3

ENV SPARK_HOME=/opt/spark
ENV PATH=$SPARK_HOME/bin:$PATH

# # Set working directory
WORKDIR ${SPARK_HOME}

# Install extra Python packages as root
USER root

RUN pip install --no-cache-dir numpy pandas smbprotocol openpyxl pyspark==3.5.1

ADD mssql-jdbc-12.10.1.jre11.jar /opt/spark/jars/
ADD ojdbc11.jar /opt/spark/jars/

## added later
RUN ln -s /opt/spark/bin/spark-submit /usr/bin/driver-py


Now Im getting this:

Error: Missing application resource.

Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]


Here is my yaml:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: juni-test
namespace: hpesbo
spec:
type: Python
sparkVersion: 3.5.1
mode: cluster
image: **********************:spark3.5-test
imagePullPolicy: Always

mainApplicationFile: "file:///maprfs-csi/test.py"

restartPolicy:
type: Never
imagePullSecrets:
- imagepull

sparkConf:
spark.mapr.user.secret: long-live-spark-secret
spark.driver.extraClassPath: "local:///maprfs-csi/snowflake-jdbc-3.13.30.jar:local:///maprfs-csi/spark-snowflake_2.12-2.12.0-spark_3.4.jar"
spark.executor.extraClassPath: "local:///maprfs-csi/snowflake-jdbc-3.13.30.jar:local:///maprfs-csi/spark-snowflake_2.12-2.12.0-spark_3.4.jar"

deps:
jars:
- local:///maprfs-csi/snowflake-jdbc-3.13.30.jar
- local:///maprfs-csi/spark-snowflake_2.12-2.12.0-spark_3.4.jar
volumes:
- name: maprfs-volume
persistentVolumeClaim:
claimName: spark-pvc
- name: rsa-key-volume
secret:
secretName: snowflake-rsa-key
- name: log-volume
persistentVolumeClaim:
claimName: spark-pvc-logs

driver:
cores: 1
coreLimit: "1000m"
memory: "8g"
labels:
version: 3.5.1
annotations:
sidecar.istio.io/inject: "false"
volumeMounts:
- name: maprfs-volume
mountPath: /maprfs-csi
- name: rsa-key-volume
mountPath: /keys
readOnly: true
- name: log-volume
mountPath: /log-csi

executor:
cores: 1
coreLimit: "1000m"
instances: 3
memory: "16g"
labels:
version: 3.5.1
annotations:
sidecar.istio.io/inject: "false"
volumeMounts:
- name: maprfs-volume
mountPath: /maprfs-csi
- name: rsa-key-volume
mountPath: /keys
readOnly: true
- name: log-volume
mountPath: /log-csi


I tried changing from mainApplicationFile: "local:///maprfs-csi/test.py" to mainApplicationFile: "file:///maprfs-csi/test.py" and mainApplicationFile: "/maprfs-csi/test.py" all of them are throwing the same error. The file is definitely present in the PVC.

I found a few issues reported in the git for the same errors but none of them have a solution. What am I missing here?

Continue reading...