I need to do the following: Check if a server is up, if so I check if the Spark cluster in that server is up, case it's down I try to start it up, case it's already up I move forward to run my Spark job. I though about creating a task to check if the Spark cluster is up (maybe trying to run a simple Spark job). If it fails I'd start the "Start Spark cluster" task. I'm using Airflow but didn't find a way to trigger a task in case the previous one fails. Besides that I'd need to check the previous one in case it succeeds so it would branch to the Spark job task and skip the "Start Spark Cluster" task. If you could provide some samples that's be great. I tried using trigger_rule and the branch operator but got nothing so far. Maybe because there's too few code examples about them on the web. Thanks Continue reading...
