[Python] Python process synchronization

I am trying to set up a number of 'worker' threads/processes, prime them with a list of 'commands' to work through and then get them to step through the commands one-by-one in step with each other.

Update I have had a few questions about why I taking this approach, so here's a bit of context: I'm using this approach for an automated testing script. I am simulating a multi-user environment, where different users are running an application that points to a shared resource. I want to carry out a sequence of operations simultaneously on an API from multiple clients. I want to be able to control what each worker does to produce a repeatable set of tests. Without the synchronisation I will have no guarantee that the operations are carried out in the order I expect. Another requirement (I possibly haven't mentioned) is that I want to have the commands executed simultaneously. E.g. All writing a large amount of data to the DB.

I am using the multiprocessing module in Python 2.7.5b3 on Windows 7. So far, I have the following example working, which is illustrative of what I am trying to do.

This example gets the workers to write results back into a shared queue, so I can see the order in which commands are executed.

The worker (worker.py):

from multiprocessing import Process, Queue, Event, Lock

class Worker(Process):

def __init__(self, execute, q_out):
Process.__init__(self)
print self.name, 'init'
self.daemon = True
self.q_in = Queue()
self.q_out = q_out
self.execute = execute

def run(self):
print self.name, 'running'
self.execute.wait()
while not self.q_in.empty():
cmd = self.q_in.get()
self.q_out.put((self.name, cmd))


The manager:

from multiprocessing import Event, Queue
from worker import Worker

if __name__ == '__main__':

workers = []
syncEvent = Event()
shared_q = Queue()
for i in range(0,2):
worker = Worker(syncEvent, shared_q)
map(worker.q_in.put, ['A', 'B', 'C'])
workers.append(worker)
worker.start()

syncEvent.set()

for w in workers:
w.join()

while not shared_q.empty():
print shared_q.get()


This gives me the output as follows:

Worker-1 init
Worker-2 init
Worker-1 running
Worker-2 running
('Worker-1', 'A')
('Worker-1', 'B')
('Worker-1', 'C')
('Worker-2', 'A')
('Worker-2', 'B')
('Worker-2', 'C')


What I am trying to achieve is this output:

Worker-1 init
Worker-2 init
Worker-1 running
Worker-2 running
('Worker-1', 'A')
('Worker-2', 'A')
('Worker-1', 'B')
('Worker-2', 'B')
('Worker-1', 'C')
('Worker-2', 'C')


I have locked at Lock andRLock, but that doesn't seem to fit the bill, as I am trying to get all the threads running at the same time, but just stop and wait until the others are all finished before executing the next command.

I am sure that there is a nice and simple way to do this, but I can't quite get my head around what it is. Does anyone have any suggestions for how to proceed?

Continue reading...