I'm currently benchmarking PySpark vs the growing alternative Polars. Basically I'm writing various queries (aggregations, filtering, sorting etc.) and measure the execution time, RAM and CPU. I intend using lazy evaluation on both ends as my use case involves larger than memory data. The problem is, to trigger the lazy query plan, I need an action like collect, count, write etc. Until now I've been doing all my test cases using .collect() for PySpark and .collect(engine="streaming") for Polars. I've read in PySpark documentation that the collect() function is inefficient for larger DataFrame outputs as it loads everything into memory. I need a "fair" way to trigger query execution that will not be disadvantageous to any of the frameworks and provide close to reality results. Should I straight up use write() to be the closest to real life use cases or is there a better way? Continue reading...
