I am attempting to combine multiple gzip streams into a single stream, my understanding is this should be possible but my implementation is flawed. Based on what I have read, my expectation was that I should be able remove the 10byte header and 8 byte footer from all streams, concatenate the bytes together and reconstruct a header and footer. However, when I do try and do this the decompression operation fails. Sample Code (Not working) import zlib import struct test_bytes = b"hello world" # create an GZIP example stream deflate = zlib.compressobj(8, zlib.DEFLATED, GZIP_WBITS) single = deflate.compress(test_bytes) single += deflate.flush() # quick sanity check that decompression works zlib.decompress(single, GZIP_WBITS) print("Single:", single.hex()) # check our understanding of the footer is correct single_len = struct.unpack("<I", single[-4:])[0] assert single_len == len(test_bytes), "wrong len" single_crc = struct.unpack("<I", single[-8:-4])[0] assert single_crc == zlib.crc32(test_bytes), "wrong crc" # Create an example GZIP stream with duplicated input bytes deflate = zlib.compressobj(8, zlib.DEFLATED, GZIP_WBITS) double = deflate.compress(test_bytes) double += deflate.compress(test_bytes) double += deflate.flush() # quick sanity check that decompression works zlib.decompress(double, GZIP_WBITS) # Check we can calculate the len and bytes correctly double_length = struct.unpack("<I", double[-4:])[0] assert double_length == len(test_bytes + test_bytes), "wrong len" double_crc = struct.unpack("<I", double[-8:-4])[0] assert double_crc == zlib.crc32(test_bytes + test_bytes), "wrong crc" print(f"Double: {double.hex()}") # Remove the header and footer from our original GZIP stream single_data = single[10:-8] print(f" Data: {' '*20}{single_data.hex()}") # Concatenate the original stream with footer removed with a duplicate # with the header and footer removed concatenated = single[:-8] + single_data # Add the footer, comprising the CRC and Length concatenated += struct.pack("<I", double_crc) concatenated += struct.pack("<I", double_length) assert concatenated .startswith(single[:-8]) print(f" Maybe: {concatenated.hex()}") # Confirm this is bad data zlib.decompress(concatenated, GZIP_WBITS) My assumption is it will be possible to use the following function to combine the crc32 values: def crc_combine(crcA, crcB, lenB): crcA0 = zlib.crc32(b'\0' * lenB, crcA ^ 0xffffffff) ^ 0xffffffff return crcA0 ^ crcB Requirements Pure python with no dependencies (aws lambda runtime where dependency management is a paid) Avoid decompressing and recompressing the streams, we have control of the original content and so can calculate the resultant CRC using the crc32_combine function. Sources Concatenate multiple zlib compressed data streams into a single stream efficiently Combining two non-pure CRC32 values Continue reading...
