[Python] How to avoid per-process memory duplication when using rapidgzip with multiprocessing...

I have a Python file:

from concurrent.futures import ProcessPoolExecutor
import tarfile, rapidgzip

def processNdjson(ndjsonName):
with rapidgzip.open(inTarDir) as myZip:
myZip.import_index(rapidgzipDir)
with tarfile.open(fileobj=myZip, mode="r:*") as f:
member = f.getmember(ndjsonName)
dataFile = f.extractfile(member)
for oneLine in dataFile:
# process oneLine here

if __name__ == "__main__":
rapidgzipDir = ...
inTarDir = ...
rapidgzipDir = ...
nCore = 5
ndjsonNames = ["name1.ndjson", "name2.ndjson"]

with ProcessPoolExecutor(nCore) as pool:
results = pool.map(worker, ndjsonNames)


Above,

• inTarDir is the directory to a .tar.gz file that contains multiple .ndjson files.
• rapidgzipDir is the pre-index file to be used by rapidgzip. This allows fast random access and is a drop-in replacement for the built-in Python gzip.GzipFile.
• Each process will

with rapidgzip.open(inTarDir) as myZip:
myZip.import_index(rapidgzipDir)
with tarfile.open(fileobj=myZip, mode="r:*") as f:


My concern: each command myZip.import_index(rapidgzipDir) will take up a certain amount of RAM (for example, 500MB for a 20GB .tar.gz file). This will grow linearly with nCore.

Is there any way to avoid the multiple import of the same rapidgzipDir?

Continue reading...