[Python] Creating Global dataset combining multiple regions in BigQuery using Apache Beam

I have four regions a, b, c d and I want to creat aa single data set concatenating all the 4 and store in c how can this be done? Tried with dbt- python but had to hard code a lot looking for a better one to go with dbt- may be apache or something Help

we did use the python module within dbt to concatenate this but had to hardcode the values as only python literals were allowed. So it is possible just need a more efficient way looking into Apache Beam maybe for some help

Continue reading...