[Python] Does gzip library in python attempt to open entire file in memory?

with gzip.open("/tar/access.tar.gz", 'rb') as f_in:
with open("/tar/access.tar", 'wb') as f_out:
shutil.copyfileobj(f_in, f_out)


My input file is 150GB. I raised my server to 432GB of ram once I realized it was running out of memory at 50GB when attempting to do this. Does gzip attempt to open the entire file in memory first? Why is 432GB not enough?

The exact error is OSError: [Errno 14] Bad address: '/tar/access.tar.gz' But this is thrown when there is a memory error.

Stack trace :

/usr/lib/python3.5/gzip.py in open(filename, mode, compresslevel, encoding, errors, newline)
51 gz_mode = mode.replace("t", "")
52 if isinstance(filename, (str, bytes)):
---> 53 binary_file = GzipFile(filename, gz_mode, compresslevel)
54 elif hasattr(filename, "read") or hasattr(filename, "write"):
55 binary_file = GzipFile(None, gz_mode, compresslevel, filename)

/usr/lib/python3.5/gzip.py in __init__(self, filename, mode, compresslevel, fileobj, mtime)
161 mode += 'b'
162 if fileobj is None:
--> 163 fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
164 if filename is None:
165 filename = getattr(fileobj, 'name', '')

OSError: [Errno 14] Bad address: '/tar/access.tar.gz'


For anyone that doesn't believe this is a memory issue and thinks the file doesn't exist, based on the poorly worded error message, with 16gb it fails within seconds, with 64GB it lasts just under a minute, and with 432GB it lasts 5 minutes before failing.

And what is the solution to uncompress a gzip file without it all going to memory?

Related question - Python OSError: Bad address when reading from large file

Also fails with the following:

with gzip.open("/dbfs/tmp/tar/access.tar.gz", 'rb') as f_in:
print("here")

Continue reading...