[Python] Can I use multi-thread to accelerate file generation by Python? what else can I do to...

I an trying to use Python to generate a compressed file. The text generation rule, i.e., pattern, is fairly naive, the overhead of computing text can be hence ignored.

I managed to put together a code to generate it in by streamline, i.e., gzip consumes the generated text on-fly, see below:

import gzip
from time import time

def gen_data( chunk_size = 1024 * 1024 ):
text = "int {n} = {n}\n"
total_size = 50 * ( 1024**3 )
n = 0
while n < total_size:
chunk = []
for _ in range( chunk_size // len( text ) ):
chunk.append( text.format( n=n ) )
n += len( text )
yield "".join( chunk ).encode( 'utf-8' )

def stream_compress():
start = time()
with gzip.open( "compressed.gz", 'wb' ) as f:
for chunk in gen_data():
f.write( chunk )
print( f"Done. Time elapsed is {time() - start:.2f} seconds" );
f.close()


However, the performance is still below my expectation, it takes ~30mins to generate a ~50G text file.

I wonder is there any way by which the performance can be yet improved?

To use multi-thread, the most intuitive way surely is to separate the file into several ones, but it is not feasible in this case due to some API restrictions, can it be applied here some way else?

Continue reading...