I have a python based service that uses libraries such as requests or curl-cffi to fetch the content of our webpages which we currently are testing for scale. As the expected response is the content of my webpage which might be around 1MB in size per request, I see an increase in the heap memory usage when a couple of requests are run which does not come down and this is much more evident at scale even after GC runs. The code snippet looks something like this: async def scrape(self, url, counter): response = None response_dict = {} try: proxy = "some proxy info" impersonate_val = "chrome" headers = {} # Setting up some headers async with AsyncSession() as s: response = await s.get( url, impersonate=impersonate_val, proxies={'http': proxy, 'https': proxy}, headers=headers, timeout=30 ) response_dict = {"response": response.text, "http_code": response.status_code} except Exception as e: response_dict = await self.scrape_url(url, counter + 1) finally: if response: await response.aclose() return response_dict Additional info: The reproducible script above is called by a ThreadPoolExecutor where each thread within the executor is associated with a long living async loop. Continuous increase in memory: Although there are no memory leaks as per tracemalloc, I found that more requests to the script led to heap memory increase over time. After deep diving, I was able to find that libraries such as cffi stores C extensions which is not cleared by python GC. So I updated my script to run the following periodically: ctypes.CDLL("libc.so.6").malloc_trim(0) Every time this runs, the memory usage shows a big dip. If there are no incoming requests, the memory comes back to its original state but, at scale, a dip is observed which increases again right after the dip to whatever level the memory was at its peak. I therefore want to understand the following: Is this behavior expected with python (According to this, this appears to be the case)? What more can I do apart from or in addition to ctypes.CDLL("libc.so.6").malloc_trim(0) to handle this leak? Continue reading...
