I was trying to run dask-distributed to distribute some big computation in a slurm cluster. I was always getting a "TimeoutError: No valid workers found" message (this came from line 6130 in distributed/scheduler.py) I have no idea how to change that Timeout parameter. I have tried in the cfg file with jobqueue.slurm.connect-timeout jobqueue.slurm.tcp-timeout jobqueue.slurm.no-workers-timeout jobqueue.slurm.death-timeout jobqueue.slurm.timeout But none of these worked. I finally had to manually modify the code to make that timeout parameter be big enough to make the scheduler wait enough to get at least one job running (the fact that the whole job raises an exception in such a short time sounds bonkers to me, but that is a different issue). So what is the config parameter to set this variable correctly in a yaml file? I have not been able to find this info in the documentation Thanks. Continue reading...
