[Python] polars - memory allocation of N bytes failed

Trying to execute a statement on a file of 30 Gbs containing 26 million records with 33 columns. When we execute on a sample with 10,000 rows, works without any errors. problem exists on the full file. Noticed that I was able to execute the code in jupyter, however this fails when running from VS Code on Windows 10 64 bit.

(
df
.group_by(person_id_key, person_source_id_key, "person_source_type")
.agg(
# All Names
pl.concat_list(
pl.col(full_name_key).fill_null(""),
pl.col(nickname_key).fill_null("").str.split("|"),
pl.col(first_name_key).fill_null("") + " " + pl.col(last_name_key).fill_null("")
)
.list.eval(pl.element().filter(~pl.element().is_in(special_removal)))
.flatten()
.unique()
.alias("name"),

# All Companies
pl.concat_list(org_name_key, master_org_name_key)
.flatten()
.unique()
.alias("company")
)
.with_columns(pl.col('name').list.len().alias("size_of_name"))
.filter(
pl.col("size_of_name") >= 1
)
.drop("size_of_name")
)


is it a problem with my script or should I try a different version of polars than 0.17.2?

Let me know in comments if there is any other information that will be helpful, the dataset is confidential but I can share a synthetic version if necessary.

Continue reading...