[Python] Pandas on spark very slow

I'm trying to learn pyspark, and for the most part I've been using the pyspark.sql methods of working with datasets. However, I was interested in the pandas on spark api as I've used pandas extensively before, but I keep facing issues with run times especially with basic stuff that I'm used to doing in pandas.

For instance, let's assume I'm working with ps_df which is a pyspark.sql dataframe and ps_pandas_df which is a pyspark.pandas dataframe:

import os
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
from pyspark.sql import SparkSession
import pyspark.pandas as ps
ps.set_option('compute.default_index_type', 'distributed')

spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "10g") \
.getOrCreate()

ps_df = spark.read.csv('/path')
ps_pandas_df = ps.read_csv('/path')

# First Example - Describing Dataset

# takes 30 seconds to run
ps_df.describe().show()

#takes 3 minutes to run
ps_pandas_df.describe()

# Second Example - Group by and aggregate

# takes around 20 seconds
temp = ps_df.groupby('A').agg(sum('B'), mean('B'))
temp.show()

# takes around 4 minutes
temp2 = ps_pandas_df.groupby('A')['B'].mean()
temp2


Also, doing some very basic stuff like .shape() seems to take forever with pandas on spark. For reference, I'm running spark on my local machine with 16 GBs of RAM and 6 cores on a dataset that's around 5GB in size with 55 million rows. If I try to do some similar stuff in dask, the performance is much better than using the pandas on spark, and I was wondering what might be causing this.

Continue reading...