I have a cluster of compute nodes, each node with many CPUs. I want them to execute commands located in a file, one command per line. The drive where the command file is located is mounted on all compute nodes. The execution times of these commands vary and to improve load balancing, I am planning to launch as many workers on each node as the number of CPUs on that node. The task of each worker would then be to retrieve one line from the command file, delete that line from the file and execute that line of command; and repeat until no more lines are left. I am planning to implement these workers with Python's multiprocessing module. The problem is that eventually multiple workers will attempt to write (i.e. remove lines) from the same file. I was wondering whether there is a simple solution to make workers wait for each other or the operating systems take care of that automatically? I believe some sort of file lock could work, however, note that there are multiple processes from multiple different compute nodes writing to the same file. Continue reading...
