[Python] Why does polars kept killing the python kernel when joining two lazy frames and...

I have one dataframe: bos_df_3 that has about a 30k+ rows and another, taxon_ranked_only, with 6 million when I tried to join them using:

matching_df = (
pl.LazyFrame(bos_df_3)
.join(
other=taxon_ranked_only,
on=["genericName", "specificEpithet", "infraspecificEpithet"],
how="right",
)
.rename({"taxonID": "matched_taxonID"})
)


And then:

matching_df.collect()

it kills the kernel of my marimo notebook. I tried to use gpu=True or streaming=True but it still crashes.
I also tried slicing taxon_ranked_only into smaller portions of length 1 million but it still crashes.
This problem seem to be similar to this:python polars numerous joins crashing

Continue reading...