[Python] Python asyncio - Download parts while extracting archives

I use Python 3.12.3/asyncio/aiohttp. I want to stick to this.

My input is a list of file parts (1 GiB each). Each file can be up to 40 parts.

[
{ "part": "file_1.7z.001" },
{ "part": "file_1.7z.030" },

{ "part": "file_N.7z.001" },
{ "part": "file_N.7z.040" },
]


I am able to do download & extract synchronously but it's inefficient:

async def main(self, files: list):
file_names = list(set([re.sub(r'\.7z.*$', '', x) for x in files) # ["file_1", "file_2"]
for file_name in file_names:
parts = [x for x in files if f"{file_names}.7z" in x]
for part in parts:
self.download(part)
self.archive_extract(file_name) # takes a lot of time blocking the for loop


async def download(self, file):
url = f"{xxx}/file"
async with self._session.get(url=url) as r:
with open(output_document, "wb") as f:
chunk_size = 64*1024
async for data in r.content.iter_chunked(chunk_size):
f.write(data)


async def archive_extract(self, file_name) -> None:
cmd = ['7z', 'x', f"{file_name}.7z.001", f"-o{save_dir}"]
subprocess.run(cmd, stderr=subprocess.STDOUT, stdout=subprocess.PIPE).stdout.decode('utf-8')


My goal is to download file parts while extracting files asynchronously:

1. Download file_N parts
2. Once done: download file_N+1 parts and start extracting file_N parts
3. and so on

How can I achieve this?

Continue reading...