I have a Polars dataframe corpus with one string column, and millions of rows. I also have a list of substrings substrings. I can take a substring and query in how many rows that substring appears with: corpus.select(pl.col('contents').str.contains(substrings[0]).sum()).item() This works well for one substring, but I have 10,000 substrings to check. What is the most efficient way in Polars to check all of them? I have considered converting substrings into its own polars dataframe, and then performing an inner-join on substring presence, grouping by keyword, then counting the size of the groups. However, this seems very expensive from a RAM overhead perspective, and I am limited on RAM. Is there a better/cleaner way? Continue reading...
