[Python] How to fairly dispatch messages to kafka?

I am building a system where multiple bots produce messages that need to be sent to Kafka. However, some bots generate many more messages than others, and I want to ensure fairness, so no single bot can flood the Kafka pipeline and starve others.

The approach I am thinking to use:

1. 
   Pushing each incoming message into a per-bot Redis list.
   
   
2. 
   Adding the bot to an activeBotIds set.
   
   
3. 
   A timer loop (setInterval) runs periodically, iterates over all active bots, pops one message per bot, sends it to Kafka, and removes empty queues from the set.
   

This ensures fairness (1 message per bot per cycle), avoids starvation, and keeps Redis operations simple.

Is there any better way to do this ?

Continue reading...