[Python] Redis connection errors on high load environment

I'm working on a very heavy load environment,

I have python code that starts a process pool of 300 processes on a server that has around 100 CPUs, and constantly runs processes that checks Redis for a value, if that value doesn't exist, it saves it and if it does, it runs some heavy calculations. it does so using the Redis get set command.

each process has a Redis client that is being initialized only once, and then used through the program

pseudo code

import redis
from concurrent.futures import ProcessPoolExecutor

redis: redis.Redis

def init_process():

global redis
redis = redis.Redis(host="127.0.0.1", port=6379, decode_response=True)

def recieved_request(unique_id):
x = redis.getset(unique_id, "hello world")
if x is not None:
heavy_cpu_load()
redis.delete(unique_id)
else:
redis.expire(unique_id, 60)

pool = ProcessPoolExecutor(max_workers=300, initializer=init_process)



def on_new_request():
pool.submit(recieved_request)




when running the code with low traffic it works as expected but when running on high traffic I get the following errors:

error 99 - connecting to cannot assign requested address

error while reading from 127.0.0.1:6379: (104, connection is reset by peer)

connection is closed by server

I've tried messing with the timeout in order to quickly close idle connections, and changing the amount of maxconnection to 50000, both lead to no changes in the behaviour.

• when running the same code without heavy CPU calculation I get no errors
• I cannot run Redis on a different machine.
• I'm using the latest Redis available (7.0.11) on centos7.8 and python3.8.10

Continue reading...