[Python] How to write GCS Parquet data to BigQuery (batch processing)

What we want to solve


Writing Parquet type data stored in GCP's CloudStorage to BigQuery tables once daily (batch processing).

I am having trouble with this implementation.

Problem


Until now, we have been storing the following CSV in GCS and running a job to write it to BigQuery using dataflow "Text Files on Cloud Storage to BigQuery " template.

Date​

Company​

Tel​

Address​

Staff​

Lank​

2022-04-01​

ABC Corporation​

00−0000−0000​

US​

mike​

0​

2022-04-04​

DEF Corporation​

00−0000−0000​

JA​

suzuki​

1​

2022-04-04​

DEF Corporation​

00−0000−0000​

UK​

john​

1​

・
・
2022-06-30​

XYZ Corporation​

00−0000−0000​

UK​

tom​

1​

About 1000 rows of data in total

Schema for each column
Dateatetime,
Company:Str,
Tel:Str,
Address:Str,
Staff:Str,
Lank:Int

Future changes


Save to GCS in Parquet instead of CSV (cannot use the above template) Write BigQuery from GCS to BigQuery once a day as a batch process.

What we want to implement


Batch processing to write Parquet type data in GCS to BigQuery table once a day.

Due to the above changes, we were unable to use our existing template, and were also troubled by the lack of other templates that we could use.

I would appreciate it if you could provide me with a GCP service (I was thinking of dataflow at the moment) that would be suitable for this job and a specific implementation method for implementing it.Thank you.

The languages used are either python or JavaScript.

Continue reading...