[Python] How can I send a .csv to a spark cluster dockerized? I have continuous path_not_found...

I'm new on apache spark and I have a problem, but I think it's more conceptual than technical.

I have created a cluster environment with a master and two workers. Then, I'm trying to execute a really simple code that create a DataFrame from a csv and show the information:

from pyspark.sql import SparkSession

#Create SparkSession
spark = SparkSession.builder \
.master("spark://127.0.0.1:7077") \
.appName("Test") \
.config("spark.driver.host", "[myIP]") \
.getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")

file= "/opt/bitnami/spark/apps/mpg.csv"
mpg_data = spark.read.csv(file,header=True,inferSchema=True)
mpg_data.show()


Then I receive this error:

Traceback (most recent call last): File "c:\spark-ml.py", line 41, in mpg_data = spark.read.csv(file,header=True,inferSchema=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\readwriter.py", line 740, in csv return self._df(self._jreader.csv(self._spark._sc._jvm.PythonUtils.toSeq(path))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\java_gateway.py", line 1322, in call return_value = get_return_value( ^^^^^^^^^^^^^^^^^ File "C:\captured.py", line 185, in deco raise converted from None pyspark.errors.exceptions.captured.AnalysisException: [PATH_NOT_FOUND] Path does not exist: file:/opt/bitnami/spark/apps/mpg.csv.
​

All the docker machines have a volume created with the route /opt/bitnami/spark/apps/ and they arrive to the file.

I think the problem is that I'm the driver and the code is executed in my computer, not in the workers. But if I point to the file in my computer, the workers cannot arrive to the file.

In a real world, what is the best way to do that? I should to create a new docker for the driver or use another system to send the file?

Thank you!

Continue reading...