[Python] Redis Broker getting OOM as "*.reply.celery.pidbox" keys count increased

Recently I have pushed around 10-15k tasks into celery by iterating in a for loop. Attaching below code for reference.

for index, htl_urls_chunk in enumerate(iteration_grouper(htl_urls, n=10, chunk_type=list)):
update_cache_for_hotel_detail_page_chunk.apply_async(kwargs={'htl_urls_chunk': htl_urls_chunk})


the length of htl_urls is around: 10-15k. Total tasks pushed ~ 1500 at given point of time.

This lead to creation of below keys in redis:

01f6ebdb-361b-32ec-afe9-2eb7c8511288.reply.celery.pidbox
6292044c-a816-349a-a159-682536ea9bfa.reply.celery.pidbox


Each key is of 150MB and total 29 keys created (4.24GB) This lead to OOM issue of my redis cluster.

Can anyone suggest why this is happening ?

I have tried to explored why these keys are being created in my redis cluster, but not able to reach any conclusion so far.

Continue reading...