[Python] launching multiple commands in parallel and parsing their outputs

I have two functions, each launch a command, and I need to run both in parallel:

def time_render(filename: str) -> str:
# Run the render and time the operation
# We expect the full path for the file
command_to_run = '{} -b {} -f 10 -- --cycles-device CUDA'.format(
render_executable, filename)
cmd = subprocess.Popen(command_to_run, shell=True, stdout=subprocess.PIPE)
elapsed_time_str = '-1' # if you see this output, something bad happened
for line in cmd.stdout:
decoded_line = line.decode().strip()
if "Time: " in decoded_line and "(Saving: " in decoded_line:
start_index = decoded_line.find(":")
end_index = decoded_line.find("(")

# this is the line we want with the render time
elapsed_time_str = convert_to_seconds(
decoded_line[start_index+1:end_index].strip())

return elapsed_time_str

def get_peak_GPU_mem() -> dict:
# Monitor the memory usage on the GPU and
# record the peak
peak_mem = {}
command_to_run = 'nvidia-smi --query-gpu=gpu_bus_id,memory.used --format=csv -l 1'
cmd = subprocess.Popen(command_to_run, shell=True, stdout=subprocess.PIPE)
for line in cmd.stdout:
decoded_line = line.decode().strip()

# The output of this command will usually look like this:
# 00000000:2D:00.0, 101 MiB <-- One GPU
# 00000000:99:00.0, 11 MiB <-- Another GPU
# Each GPU will have a unique PCI ID
split_tokens = decoded_line.split(',')
gpu_id = split_tokens[0].strip()
curr_mem = int(split_tokens[1].strip().split()[0])
if gpu_id not in peak_mem:
peak_mem[gpu_id] = curr_mem
else:
# we have a previous entry, compare the entries
if curr_mem > peak_mem[gpu_id]:
peak_mem[gpu_id] = curr_mem


I get that I could probably use the multiprocessing module to run these in parallel, but I need to be able to terminate the get_peak_GPU_mem function since that command never terminates. How can I terminate it such that it returns the value it's building? Is there a better way to organize this? I basically need to run those two commands and monitor their respective outputs in specific ways.

Edit: To clarify, I need to be able to terminate a function while getting a value back since the command it runs never stops.

Continue reading...