[Python] Streaming the reading of a very large **compressed** json file in Python

I have a very large (too large to hold in RAM) .json.zstd file that I built iteratively with a generator of texts data_chunks. A completely toy example of such generator is [{"text": "very very"}," very very", " very very very", {"text": " very very long"}] obviously here in this toy example it can hold in RAM.

import zstandard as zstd

# real code used to write the very large file
def write_to_file(output_path, data_chunks, level=22):
cctx = zstd.ZstdCompressor(level=level)

with open(output_path, 'wb') as f_out:
with cctx.stream_writer(f_out) as compressor:
for idx, chunk in enumerate(data_chunks):
if isinstance(chunk, str):
pass
elif isinstance(chunk, dict):
chunk = chunk["text"]
else:
raise ValueError(f"Unrecognized chunk {type(chunk)}")
if idx == 0:
chunk = '{"text": "' + chunk
chunk = chunk.encode("utf-8")
compressor.write(chunk)
compressor.write('"}'.encode("utf-8"))

# toy example
write_to_file("test.json.zstd", [{"text": "very very"}," very very", " very very very", {"text": " very very long"}], level=22)


This compressed file now just holds a json file {"text": "very very very long text ... "} We can convince ourselves of this by reading it easily:

import io
def read_zstd_lines(input_path):
dctx = zstd.ZstdDecompressor()
with open(input_path, 'rb') as compressed:
with dctx.stream_reader(compressed) as decompressor:
text_stream = io.TextIOWrapper(decompressor, encoding='utf-8')
for line in text_stream:
if line.strip():
yield json.loads(line)
next(read_zstd_lines("test.json.zstd"))


I am looking for a solution similar to ijson that would read it in RAM iteratively by (potentially size-adjustable) chunks (or just chunks no larger than a fixed size) but that would work on the compressed file as the example above.

Continue reading...