[Python] How to run few parallel instances of a python infinite loop bot with different...

I have one year of experience with Python; however, I still consider myself a beginner because I lack a solid understanding of its fundamentals.

I created a bot which executes trades on forex exchange. Now, I need to run few instances of this bot simultaneously, each one for an individual forex pair (e.g. 'EUR_USD', 'USD_JPY'). I kindly want to ask for tips how to do this properly and reliably?

My bot has few states like:

1. Requests exchange and get recent data once in 5 mins
2. Checks for signal to open position
3. If signal exists, send request to open position
4. Watches open position until it's closed
5. Starts the whole process again

Here is the code snippet from core program

def run(self):

if self.state == 'fetch_data':
self.s1.run_pending()

elif self.state == 'check_signal':
self.check_signal()

elif self.state == 'open_position':
self.open_position()

elif self.state == 'watch_trade':
self.watch_trade()

I need this bot to run in background for many days. So in my main.py I do this:


def run_bot(ticker):
bot = Bot(ticker)
while True:
bot.run()
time.sleep(0.1)

Then, I tried to use multiprocessing (and concurrent.Futures) to run few instances at the same time.


from concurrent.futures import ProcessPoolExecutor

if __name__ == '__main__':

with ProcessPoolExecutor(max_workers=3) as executor:
executor.map(run_bot, TICKERS)
print(f'Run process ...')

The problem with multiprocessing is that it runs for sometime and then stuck (approximately after 1 hour). Basically, it starts to skip some requests scheduled. Sometimes it resumes later and can make few requests (usually with delay of few minutes compared to scheduled time) and then stucks again.

I did some research and understood that this is existing issue with multiprocessing. There is a fix proposed here. I tried it, but it didn't work for me. It throws error because of this line of code. However, this line was actually proposed as a solution.


from multiprocessing import set_start_method
set_start_method("spawn")

To sum up, there are few questions I'm trying to find answers to:

1. A general one. What is the best practice to run few instances of an app / bot simultaneously?
2. Is multiprocessing an option for me at all(?) taking into account that my bot is an infinite loop. The process never ends until is stopped.
3. What are the potential alternatives?

Note: it's important for me that each of the bot instances to be kinda independent. If one instance throws error or crashes other should keep running.

Continue reading...