[Python] My asynchronous program freezes when trying to read the html of the response

I recently started learning asyncio in python and faced a problem when I was practicing Queues. My task was to write a code that gets random pics from site, parses img links and downloads those images asynchronously using Queues. Unfortunately, my programm just freezes trying to read html of a response and I have no idea what to do with it. Here is my code:

async def make_request(url, session):
async with session.get(url) as response:
if response.ok:
return response
else:
print(f'{url} returned: {response.status}')


async def get_image_page(queue, session):
url = "https://c.xkcd.com/random/comic/"
response = await make_request(url, session)
await queue.put(str(response.url))

def parse_link(html):
soup = BeautifulSoup(html, 'lxml')
u = soup.select_one('div#comic>img').get('src')
image_url = f'https:{u}'
return image_url


async def get_image_url(pages_queue, image_urls_queue, session):
while True:
url = await pages_queue.get()
response = await make_request(url, session)
html = await response.text()

loop = asyncio.get_running_loop()
with ProcessPoolExecutor() as pool:
image_url = await loop.run_in_executor(
pool, parse_link, html
)

await image_urls_queue.put(image_url)

pages_queue.task_done()


async def download_image(url):
pass

async def main():
async with aiohttp.ClientSession() as session:
pages_queue = asyncio.Queue()
image_urls_queue = asyncio.Queue()

page_getters = []
for _ in range(4):
task = asyncio.create_task(
get_image_page(pages_queue, session)
)
page_getters.append(task)

url_gettes = []
for _ in range(4):
task = asyncio.create_task(
get_image_url(pages_queue, image_urls_queue, session)
)
url_gettes.append(task)

await asyncio.gather(*page_getters)
await pages_queue.join()

for task in page_getters:
task.cancel()

print(image_urls_queue)


if __name__ == '__main__':
asyncio.run(main())


I tried to debug get_image_url() coroutine and figured that problem is in line 36

html = await response.text()


Can someone point out to me what I did wrong and how can I fix this?

Continue reading...