I'm building a beam.DoFn in Python 3.9 using FROM gcr.io/dataflow-templates-base/python3-template-launcher-base with beam version 2.64.0. In my Dockerfile I have these lines: COPY binary /app/amtoolkit/ ENV PATH="/app/amtoolkit:${PATH}" RUN chmod +x /app/amtoolkit/binary Inside my Python file, in the process() method of my DoFn, I have these lines: cwd = "/app/amtoolkit" cmd = [ "./binary"] self.subprocess.run(cmd, check=True, stdout=self.subprocess.PIPE, stderr=self.subprocess.PIPE, cwd=cwd) When I run the pipeline using a local launcher, everything works great. When I try to run this job as a Flex Template, the subprocess.run command fails with an error: [Errno 2] No such file or directory: '/app/amtoolkit' I've added all the reasonable logging I can think of to try to figure out where this directory has gone, but to no avail. Is there any known issue with Flex Templates that would cause a directory from an image to go missing? Could it be possible the execution is being distributed to a worker that does not have the specified image? I know that imports have to be added inside function bodies, for example. Is there a corresponding gotcha for binaries included in the image? I've tried logging at every phase of launch from the top level of the .py file through main, DoFn.init and DoFn.process. The directory does not exist when run as a Flex Template, but it does exist when run with a local launcher. I have verified that the exact same image is being used in both cases. I saw this question and answer which indicate that the workers do not use my container (weird), but no answer to the follow up question about how to specify the image for the workers. How can I get my binary to the workers for the subprocess.run call? Continue reading...
