I have a s3 location in which I have a list of directories and each directory contains a csv named sample_file.csv. I am trying to read these files using a glob pattern in pl.read_csv but it is just reading only one file and silently ignoring the rest. The issue has been mentioned in the polars git (link) issues earlier but there it seems to have been resolved and I want to understand in case I am doing something wrong. Below is my code import polars as pl s3_bucket = "sample_bucket" prefix = "abcd/efgh/ijkl/" storage_options = {"expand": True} df = pl.read_csv(source = f"s3://{s3_bucket}/{prefix}*/*.csv", storage_options = storage_options) I can change df = pl.read_csv(source = f"s3://{s3_bucket}/{prefix}*/*.csv", storage_options = storage_options) to df = pl.read_csv(source = f"s3://{s3_bucket}/{prefix}*/sample_csv.csv", storage_options = storage_options) and nothing changes I have checked and only the sample_csv.csv from the first directory present at prefix has been read in the df scan_csv seems to work just fine import polars as pl s3_bucket = "sample_bucket" prefix = "abcd/efgh/ijkl/" df = pl.scan_csv(source = f"s3://{s3_bucket}/{prefix}*/*.csv").collect() without the storage_options the read_csv would not expand the glob and throw filenotfound error but here in the case of scan_csv even that is not needed. Weird tribal knowledge of polars if there is no catch to this and I am not doing anything wrong!! What am I doing wrong? Continue reading...
