In my AWS Glue (4.0 which supports spark 3.3), I am trying to optimize by using this: spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true") but it gives me a warning /opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:425: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below: PyArrow >= 1.0.0 must be installed; however, it was not found. Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true. warn(msg) I have ensured that I have added pyarrow (explicitly to glue job) to s3 bucket s3://your-bucket/libs/pyarrow-12.0.1-cp37-cp37m-manylinux2014_x86_64.whl and added it to Glue Job > Job Details > Python library path Continue reading...
