I recently discovered the Spark Connect new feature in Spark 3.4. I have tested it on a YARN cluster and it works well! However, anyone can access my Spark Connect server and run jobs with my user. Is it possible to configure authentication for Spark Connect? A simple password, or the combination of an user name and a password. I search something like: Server side ./sbin/start-connect-server.sh \ --jars jars/spark-connect_2.12-3.4.1.jar \ --master yarn --name SparkConnectTest \ --conf spark.sql.catalogImplementation=hive \ --conf password=mysuperpassword Client side spark = SparkSession.builder.remote("sc://localhost").conf("password", "mysuperpassword").getOrCreate() The official documentation says: While Spark Connect does not have built-in authentication, it is designed to work seamlessly with your existing authentication infrastructure. Its gRPC HTTP/2 interface allows for the use of authenticating proxies, which makes it possible to secure Spark Connect without having to implement authentication logic in Spark directly. But I don't know how to configure gRPC with Spark Connect. Python 3.7.6 PySpark 3.4.1 Hadoop 3.1.1 Continue reading...
