My python application is continuously running in K8s pod in AWS environment over Alpine based linux OS. It is consuming data from kafka, storing it in a queue. Another thread gets the data from queue and starts multiprocessing (5 parallel process). Each worker process does some preprocessing(calcualtion stuff over dataframes and storing into database) over their respective dataframes and return to the main process after doing their job. Now the problem is memory usage is increasing continuously inside pod when I do kubectl top pod. In windows, in pycharm I don't see such problem. Mostly it involves dataframe creation and manipulation. There is one global cache that is also a dataframe. I've tried these so far, no luck so far: 1.) Deleted dataframes after their use. 2.) gc.collect() 3.) malloc_trim() not working in alpine based os I have assigned 5 GB to each pod (2 pod running). I am not sure why this issue is coming. This is life or death for me. Someone save me!!! Continue reading...
