Suppose I have two dataframes, small_df and huge_df in two parquet files. small_df has about 500K rows and huge_df has more than 10 billion lines. They both have a column "A", and small_df is not sorted on "A", but huge_df is. I want to sort small_df and left-join it with huge_df on "A" in the fastest possible way. I have a lot of memory available and a lot of CPUs/GPUs. How can I do it in Python? My current approach: scan both as lazyframes in Polars, sort small_df and join it with huge_df, but I am not sure if I should filter first or what engine to use or if there are better libraries at all. Continue reading...
