[Python] Can child processes create shared memory and share it with parent processes?

Can a child process create a SharedMemory object and share it with a parent process? Currently I am getting errors when I try. I need this because creating and copying memory is the major performance bottleneck in my application, but I need to write the shared memory in the parent process.

Here are three minimal examples illustrating the issue. I use time.sleep(1) to control whether the parent or child process completes first.

Case 1 (no error): Create shared memory in parent process, share with child

import time
from multiprocessing import shared_memory, Process
def worker():
shm = shared_memory.SharedMemory(name="test")

shm = shared_memory.SharedMemory(name="test", create=True, size=1)
Process(target=worker).start()
time.sleep(1)
shm.unlink()


Case 2 (no error): Creating shared memory in child process, not sharing with parent

import time
from multiprocessing import shared_memory, Process
def worker():
shm = shared_memory.SharedMemory(name="test", create=True, size=1)
time.sleep(1)
shm.unlink()

Process(target=worker).start()


Case 3 (ERROR): Create shared memory in child process, share with parent

import time
from multiprocessing import shared_memory, Process
def worker():
shm = shared_memory.SharedMemory(name="test", create=True, size=1)
time.sleep(1)
shm.unlink()


Process(target=worker).start()
shm = shared_memory.SharedMemory(name="test")


Traceback:

Traceback (most recent call last):
File "/home/benjamin/Documents/duckit/covin.py", line 24, in <module>
shm = shared_memory.SharedMemory(name="test")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/benjamin/miniforge3/envs/hichdev/lib/python3.12/multiprocessing/shared_memory.py", line 104, in __init__
self._fd = _posixshmem.shm_open(
^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/test'


Motivation:

I'm trying to use numpy loadtxt in a numer of child processes over a bunch of files (a CPU-bottlenecked, not I/O-bottlenecked task, hence being done with multiprocessing). Then I want to write the arrays to an HDF5 file, which is most efficient when done sequentially in the main thread. If the arrays must be copied for the parent process to receive them (as when passing then via a multiprocessing queue or returning them), then this becomes the performance bottleneck for the whole program. Hence, I want to copy the arrays into shared memory in the child processes and share them with the parent process in order to write them.

Continue reading...