[Python] How to read multiple parquet files from a gcp storage bucket using duckdb and python

• python version: Python 3.12.3
• duckdb version: "1.1.3"

I'm trying to get duckdb working to read multiple parquet files from a storage bucket dir using a glob pattern.

I can load a single file using the following:

# works
duckdb.register_filesystem(filesystem("gcs"))
test_file = "gcs:///some_bucket/some_dir/some_file.parquet"
data = duckdb.sql(f"SELECT * FROM read_parquet('{test_file}')")


Based on https://duckdb.org/docs/data/multiple_files/overview.html I expect to be able to use the following, but it fails:

# fail
duckdb.register_filesystem(filesystem("gcs"))
test_file = "gcs:///some_bucket/some_dir/*.parquet"
data = duckdb.sql(f"SELECT * FROM read_parquet('{test_file}')")


With the error:

duckdb.duckdb.IOException: IO Error: No files found that match the pattern "gcs:///some_bucket/some_dir/*.parquet"


I know that there is a parquet file which would match that glob though, based on the initial (working) version which explicitly states the file to query.

Loading multiple via a list


Note, when I tried the following:

a = "gs://some_bucket/some_dir/a.parquet"
b = "gs://some_bucket/some_dir/b.parquet"
data = duckdb.sql(f"SELECT * FROM read_parquet(['{a}', '{b}'], union_by_name=true)")


It works, though I'm not sure if this is relevant to my initial issue re globs or not.

Edit: ListObjectV2 required for file glob


Following answer:

> The documentation for the httpfs extension implies that globbing is only supported for s3. See https://duckdb.org/docs/extensions/httpfs/overview.html

I was confused as (to me) the docs here: https://duckdb.org/docs/extensions/httpfs/s3api seemed to imply that google storage was also supported, looking again (at https://duckdb.org/docs/extensions/httpfs/s3api ) it states:

File glob ListObjectV2


Searching for support on this brings up:

• (7 years old) Is the ListObjectsV2 api call implemented in Google Cloud Storage? and the
• https://issuetracker.google.com/issues/155109631

At the end of the https://issuetracker.google.com/issues/155109631 thread there's a link to: https://cloud.google.com/storage/docs/release-notes#July_1k2_2021 which states:

List object V2 for the XML APIPreview launched.

List object V2 provides improved interoperability with Amazon S3 tools and libraries.


I don't know enough about this to know if that's what duckdb needs.

Continue reading...