[Python] Polars running out-of-memory when scanning over large amount of IPC files

I have a large amount of IPC files that all contain the same columns.

The data in this files is basically a unique ID and a large string (JSON string). What I would like to do is selecting specific JSON strings based on a list of given IDs. However, when using scan_ipc over the folder, and then filtering and selecting the respective IDs and columns I run OOM on collect()

What i tried is the following:

ids = [...]
df = pl.scan_ipc("<path/to/folder>")
df = df.filter(pl.col("ID").is_in(ids))
df = df.select(pl.col("JSON"))
df.collect()


I have found a workaround by first looking up the individual files and then selecting from these. But this seems a bit convoluted and defeats the purpose of streaming in my opinion.

def get_files(ids):
i = pl.scan_ipc("<path/to/folder>", include_file_paths="files")
i = i.filter(pl.col("ID").is_in(ids))
i = i.select(pl.col("files")).unique()
return i.collect()["files"].to_list()

ids = [...]
results = []
for file in get_files(ids):
results.append(
pl.scan_ipc(file)
.filter(pl.col("ID")
.is_in(ids))
.select(pl.col("JSON"))
.collect()
)

pl.concat(results)


It is very likely that I am doing something wrong here so I would be very happy over any suggestion on how this could be done more efficiently.

Thanks in advance!

Continue reading...