[Python] How do I know if my `concurrent.futures.ThreadPoolExecutor` is actually running in...

I am trying to parallelize a for loop that appends to a list using Python using the concurrent.futures.ThreadPoolExecutor class. My code works, but I'm not sure if it's actually doing multithreading. According to Python's documentation here, the .map(func, iterable) method of Executor instances should call the function over each element of the iterable in a somewhat arbitrary order:

map(func, *iterables, timeout=None, chunksize=1)

Similar to map(func, *iterables) except:

- the iterables are collected immediately rather than lazily;

- func is executed asynchronously and several calls to func may
be made concurrently.


But the output of my code appears in the same order as the iterable, which makes me think it isn't actually running in parallel:

from concurrent.futures import ThreadPoolExecutor
import itertools


def foo(xy):
x, y = xy
return f"{x} + {y} = {x + y}"


if __name__ == "__main__":
n = 8
with ThreadPoolExecutor(4) as e:
prod = itertools.product(range(0, n), range(n, 2 * n))
lst = list(e.map(foo, prod))
assert len(lst) == n * n # make sure we got them all
print(lst)


The output is

> python main.py
['0 + 8 = 8', '0 + 9 = 9', '0 + 10 = 10', '0 + 11 = 11', '0 + 12 = 12', '0 + 13 = 13', '0 + 14 = 14', '0 + 15 = 15', '1 + 8 = 9', '1 + 9 = 10', '1 + 10 = 11', '1 + 11 = 12', '1 + 12 = 13', '1 + 13 = 14', '1 + 14 = 15', '1 + 15 = 16', '2 + 8 = 10', '2 + 9 = 11', '2 + 10 = 12', '2 + 11 = 13', '2 + 12 = 14', '2 + 13 = 15', '2 + 14 = 16', '2 + 15 = 17', '3 + 8 = 11', '3 + 9 = 12', '3 + 10 = 13', '3 + 11 = 14', '3 + 12 = 15', '3 + 13 = 16', '3 + 14 = 17', '3 + 15 = 18', '4 + 8 = 12', '4 + 9 = 13', '4 + 10 = 14', '4 + 11 = 15', '4 + 12 = 16', '4 + 13 = 17', '4 + 14 = 18', '4 + 15 = 19', '5 + 8 = 13', '5 + 9 = 14', '5 + 10 = 15', '5 + 11 = 16', '5 + 12 = 17', '5 + 13 = 18', '5 + 14 = 19', '5 + 15 = 20', '6 + 8 = 14', '6 + 9 = 15', '6 + 10 = 16', '6 + 11 = 17', '6 + 12 = 18', '6 + 13 = 19', '6 + 14 = 20', '6 + 15 = 21', '7 + 8 = 15', '7 + 9 = 16', '7 + 10 = 17', '7 + 11 = 18', '7 + 12 = 19', '7 + 13 = 20', '7 + 14 = 21', '7 + 15 = 22']


which is ordered exactly the same way as the product iterator.

Compare, for instance, the following Julia code:

using Base.Threads

foo(x, y)::String = "$x + $y = $(x + y)"

function main()
vec = String[]
n = 8
tuples = Iterators.product(1:n, n+1:2n) |> collect
@threads for (x, y) in tuples
push!(vec, foo(x, y))
end
@assert length(vec) == n * n
display(vec)
end

main()


The output is not in the same order as the iterator, which implies that the code was executed in parallel—which is what I want:

> julia --threads=auto main.jl
64-element Vector{String}:
"1 + 9 = 10"
"2 + 9 = 11"
"3 + 9 = 12"
"4 + 9 = 13"
"5 + 9 = 14"
"6 + 9 = 15"
"3 + 13 = 16"
"4 + 13 = 17"
"5 + 13 = 18"
"6 + 13 = 19"
"7 + 13 = 20"
"5 + 10 = 15"
"6 + 10 = 16"
"7 + 10 = 17"
"8 + 10 = 18"
"1 + 11 = 12"
"2 + 11 = 13"
"5 + 14 = 19"
"6 + 14 = 20"
...

Questions

• How do can I tell if my Python code is actually running in parallel?
• If it isn't, how can I make it so?

This question shows how to obtain similar behavior to Julia's @threads using multiprocessing.dummy import Pool, but I am trying to learn the concurrent.futures.ThreadPoolExecutor class, hence the new question.

Continue reading...