In Dataflow, you can run jobs without autoscaling. This is typically achieved by setting a pipeline_option called autoscaling_algorithm to NONE. Attempting the equivalent on Templated Dataflow Jobs seems to fail, as this is not considered a templated option. Moreover, other settings such as setting the HD type to an SSD "worker_disk_type": f"compute.googleapis.com/projects/{GCP_PROJECT_ID}/zones/us-central1/diskTypes/pd-ssd", also seem to fail as these pipeline options are not recognized within Classic Dataflow Templates. What is their equivalent syntax? I have attempted to use the equivalent properties and similar looking ones too however they have failed to trigger my pipeline. Continue reading...
