I am looking for insights into how you may have tackled a similar problem or solution or if you may have any ideas or suggestions. I have a Python application that does a basic job: Pull message from Kafka topic A -> Run algorithm on datapoint -> Increment a redis counter + push to Kafka topic B. Constraints: The message count is 5+ Million a day and each data point takes ~1.5 second to process. Current strategy: Deploy the service with x cores. Have 3 asyncio routines in each pod: (1) that pull the data and submits to inbound asyncio queue, (2) that pulls for queue, submits batches of data to ProcessPool, gets a callback once done and submits to an outbound asyncio queue, (3) that pulls messages from the outbound queue, increments Redis, pushes output to Kafka and commits latest offset. I am facing an issue here: The number of workers I am able to provide for ProcessPool are always 1-2 less than the total cores available as the main async routine is using upto 2 cores. Even then it is throttling in Kubernetes so I am not able to squeeze out 100% performance. Any suggestions on improvements or change of strategy are appreciated! Continue reading...
