I have an Python script which uses asyncio to make over a million requests. I first ran into memory issues and then discovered semaphores, I have since implemented a semaphore successfully to limit the number of concurrent task as well as tasks in the queue at one time. My program loads a list of requests like this: with open(wordlist) as words: w = words.read().splitlines() Then that list gets passed to the following function for processing and the actual work gets done. async def _process_dns_wordlist(self, wordlist, domain): """Takes a list of words and adds them to the task list as space is available""" for word in wordlist: # Wait on the semaphore before adding more tasks await self.sem.acquire() host = '{}.{}'.format(word, domain) task = asyncio.ensure_future(self._dns_lookup(host)) task.add_done_callback(functools.partial(self._dns_result_callback, host)) self.tasks.append(task) await asyncio.gather(*self.tasks, return_exceptions=True) Before I implemented the semaphore the program would just crash, running out of memory when I was queueing up all the tasks, now it runs a while and then crashes because it runs out of memory about 1/2 way through the requests. I assume that this is because after the future is processed by my callback it sits there in memory wasting space. My problem is I cannot figure out what to use to delete the processed future once I'm done with it. I've read the asyncio docs and I don't see a destroy/delete method. Am I missing something really obvious? Thanks for your help! Continue reading...
