[Python] How to Set Dask Dashboard Address with SLURMRunner (Jobqueue) and Access It via SSH...

I am trying to run a Dask Scheduler and Workers on a remote cluster using SLURMRunner from dask-jobqueue. I want to bind the Dask dashboard to 0.0.0.0 (so it’s accessible via port forwarding) and access it from my local machine.

However, the dashboard always binds to the default IP and port (10.x.x.x:8787), and none of my attempts to configure it seem to work. Here’s what I’ve tried so far:

1. Environment Variable Approach


Before running the script, I set:

export DASK_DISTRIBUTED__SCHEDULER__DASHBOARD__ADDRESS="0.0.0.0:8789"
python my_script.py


But when I check the logs, the dashboard is still bound to the default IP and port (8787).

2. Programmatically Setting the Dashboard Address


I tried setting the dashboard address using dask.config.set() before initializing the SLURMRunner:

import dask
from dask_jobqueue import SLURMRunner
from dask.distributed import Client

dask.config.set({"distributed.scheduler.dashboard.address": "0.0.0.0:8789"})

runner = SLURMRunner()

client = Client(runner)

print(client.dashboard_link)


Despite setting this, the logs still show the dashboard binding to 10.x.x.x:8787.

3. Using a Configuration File


I created a dask.yaml file at ~/.config/dask/dask.yaml with the following content:

distributed:
scheduler:
dashboard:
address: "0.0.0.0:8789"


I ensured the configuration was being picked up by running:

but the scheduler still binds to 10.x.x.x:8787.

Question:


How can I force the Dask Scheduler launched by SLURMRunner to bind the dashboard to 0.0.0.0:8789 so that I can access it via SSH port forwarding?

Is there another way to configure or override this behavior in SLURMRunner?

Notes:

• Python 3.10.10, dask 2024.9.1, dask-jobqueue 0.9.0
• SLURMRunner is part of a larger script, so I’d prefer to stick with it rather than switching to manual dask-scheduler launches.

Any help or suggestions would be greatly appreciated!

Continue reading...