Let's say I have two batches of tasks to run with Dask. The batches will be sent to the same scheduler (a different machine with many worker machines) and will be submitted from different clients (different users on different machines). They will have the same priority value. When a processing slot frees up, I want Dask to consider every task across both batches, then randomly choose one based on priority value alone. I do not want submission time to be considered. I want tasks across every batch to be given an equal chance to run when the priority value is equal. I've tried modifying the fifo_timeout but that just seems to switch the behavior from FIFO to LIFO. In both cases, one batch is run in its entirety, then the other one runs. I've looked into scheduler plugins but I don't see any examples of plugins that change task ordering behavior. I've also looked into modifying the Dask source code but I'm not sure where to begin. How should I approach this problem? Continue reading...
