[Python] Worker cannot find external file in Apache Beam

I have a simple function that reads from Mongo using Apache Beam

def create_mongo_pipeline(p: beam.Pipeline, mongo_uri: str, db: str, coll: str, cert_file: str, gcs_bucket: str) -> None:
# Read from MongoDB
docs = p | 'ReadMyFile' >> beam.io.mongodbio.ReadFromMongoDB(
uri=mongo_uri,
db=db,
coll=coll,
bucket_auto=True,
extra_client_params={
"tls": True,
"authMechanism": "MONGODB-X509",
"tlsCertificateKeyFile": cert_file,
}
)
...


cert_file is defined in a config.json

{
"connection":
{
"type": "mongo",
"uri": "mongodb://blablabla:27017",
"db": "mydb",
"certificate": "resources/mycert.pem"
}
}


Everything runs smoothly in local (with the Direct Runner). However as soon as I run the script in Dataflow I get the following error

RuntimeError: FileNotFoundError: [Errno 2] No such file or directory: 'resources/mycert.pem' [while running 'ReadMyFile/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction-ptransform-37']


I have tried packaging all the code in a Docker container but I keep on getting the same error.

Just for the sake of completeness, this is how I call the Pipeline when using the Dataflow runner

beam_options = PipelineOptions(
runner="DataflowRunner",
project=config["project_id"],
job_name=config["job_name"],
experiments=["use_grpc_for_gcs"],
temp_location=f"{config['gcs_bucket']}/temp",
)

Continue reading...