[Python] Loading BigQuery tables from large pandas DataFrames

I am trying to load a relatively large pandas dataframe df into a Google BigQuery table table_ref using the official python google-cloud-bigquery client library.

So far I have tried two different approaches:

1) load the table directly from the dataframe in memory

client = bigquery.Client()
client.load_table_from_dataframe(df, table_ref)


2) save the dataframe to a parquet file in Google Cloud Storage at the uri parquet_uri and load the table from that file:

df.to_parquet(parquet_uri)
client = bigquery.Client()
client.load_table_from_uri(parquet_uri, table_ref)


Both approaches lead to the same error:

google.api_core.exceptions.BadRequest: 400 Resources exceeded during query execution: UDF out of memory.; Failed to read Parquet file [...]. This might happen if the file contains a row that is too large, or if the total size of the pages loaded for the queried columns is too large.
​

The dataframe df has 3 columns and 184 million rows. When saved to parquet file format, it occupies 1.64 GB.

Is there any way to upload such a dataframe into a BigQuery table using the official python client library?

Thank you in advance,

Giovanni

Continue reading...