[Python] Binding Python process to specific cores (Linux) using mpirun

I have a Python file bla.py:

import os
from mpi4py import MPI
import psutil

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
PID = os.getpid()
cpu_affinity = psutil.Process().cpu_num()

print(f'rank: {rank} has PID: {PID} with affinity {cpu_affinity}')


But when I execute it using mpirun --cpu-set 15-20 --bind-to core -n 6 python3 bla.py, I get:

rank: 5 has PID: 2451954 with affinity 16
rank: 2 has PID: 2451923 with affinity 15
rank: 0 has PID: 2451911 with affinity 20
rank: 4 has PID: 2451944 with affinity 18
rank: 3 has PID: 2451935 with affinity 16
rank: 1 has PID: 2451919 with affinity 17


Note how there are 2 processes with affinity 16 and no process with affinity 19. This is non-deterministic and sometimes the processes are actually mapped 1:1 to specific cores, and sometimes not.

I tried different combinations of --bind to and --map-by but I wasn't able to get it to properly work.

With --display-map:

Data for JOB [62540,1] offset 0 Total slots allocated 24

======================== JOB MAP ========================

Data for node: triton Num slots: 24 Max slots: 0 Num procs: 6
Process OMPI jobid: [62540,1] App: 0 Process rank: 0 Bound: UNBOUND
Process OMPI jobid: [62540,1] App: 0 Process rank: 1 Bound: UNBOUND
Process OMPI jobid: [62540,1] App: 0 Process rank: 2 Bound: UNBOUND
Process OMPI jobid: [62540,1] App: 0 Process rank: 3 Bound: UNBOUND
Process OMPI jobid: [62540,1] App: 0 Process rank: 4 Bound: UNBOUND
Process OMPI jobid: [62540,1] App: 0 Process rank: 5 Bound: UNBOUND

=============================================================


With --report-bindings:

[triton.ecn.purdue.edu:2486163] MCW rank 0 is not bound (or bound to all available processors)
[triton.ecn.purdue.edu:2486163] MCW rank 1 is not bound (or bound to all available processors)
[triton.ecn.purdue.edu:2486163] MCW rank 2 is not bound (or bound to all available processors)
[triton.ecn.purdue.edu:2486163] MCW rank 3 is not bound (or bound to all available processors)
[triton.ecn.purdue.edu:2486163] MCW rank 4 is not bound (or bound to all available processors)
[triton.ecn.purdue.edu:2486163] MCW rank 5 is not bound (or bound to all available processors)


How do I pin my 6 launched processes to 6 different cores?

Continue reading...