[Python] Why does this Dataflow pipeline run with no errors, but doesn't read any data?

I'm running a streaming Dataflow pipeline from MongoDB to BigQuery. This is the code I'm using:

beam_options = PipelineOptions(
beam_args,
runner='DataflowRunner',
project=my_project,
job_name=my_job_name,
temp_location=my_temp_location,
region=my_region,
streaming=True,
subnetwork=my_network,
service_account_email=my_sa)


with Pipeline(options=beam_options) as p:
input = p | "Read from MongoDB" >> ReadFromMongoDB(uri=c_string, db=my_db, coll=my_coll, filter=my_filter, bucket_auto=True)
input | "Write to BigQuery" >> WriteToBigQuery(table=my_table, dataset=my_dataset, project=my_project, method='STREAMING_INSERTS')



The pipeline starts smoothly and doesn't show any errors in the job page. However, it doesn't read any data or create any workers. The number of workers and CPU processed remains 0.

On looking at the GCE logs, I can see the following error:


The resource 'projects/my_project/global/instanceTemplates/my_instance_template' already exists


It is repeated every minute, and I've also seen a code error 13 in similar logs. I'm not sure what's going on, but it seems Dataflow is having issues spinning up the necessary infrastructure for the job to run.

Any idea on what could be causing this behaviour?

NB: I have omitted all the PII due to privacy concerns.

Continue reading...