Is there any way to plot information from Spark dataframe without converting the dataframe to pandas? Did some online research but can't seem to find a way. I need to automatically save these plots as .pdf, so using the built-in visualization tool from databricks would not work. Right now, this is what I'm doing (as an example): # df = some Spark data frame df = df.toPandas() df.plot() display(plt.show()) I want to produce line graphs, histograms, bar charts and scatter plots without converting my dataframe to pandas dataframe. Thank you! Continue reading...
