I have one websocket that processes incoming information and distributes received information to other queues. This websocket works in single-threaded mode. This websocket sorts information per title. I need to create about 100-200 parallel threads or Linux processes inside of which the incoming information will be processed. Inside the thread, a thread-safe queue of information that processes in turn. Incoming information will be every second for each title. On one side, data is added, on the other, each second is taken, added to an array (about 1800 rows) and this array is processed. If it passes the filter, then the data goes to the server. I understand that I need to work with Queue. Total: 1 websocket receives data on 100-200 headers. The websocket distributes data by header and sends it to its threads Each thread has its own parallel processing of incoming information Inside the thread, new information is added to the Queue, data is taken from this queue, added to the array (1800) and processed. After processing, the oldest lines are deleted and new ones are taken from the queue. The difficulty is how to parallelize correctly so as not to lose the speed of data processing, because it is very important? What VPS is needed for this? Is it possible to use django for such case? I don't fully understand how exactly to work with threads in my case. Continue reading...
