[Python] How to avoid interleaved logs in parallel processing?

Naively parallelizing some tasks that involve logging results in interleaved logs:

import logging
from concurrent.futures import ThreadPoolExecutor
from time import sleep


logger = logging.getLogger(__name__)


def task(n):
logger.info(f"Task {n} started")
sleep(0.001)
logger.info(f"Task {n} finished")
return n * n


def main():
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(task, i) for i in range(5)]
_ = [f.result() for f in futures]


if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
main()

---

INFO:__main__:Task 0 started
INFO:__main__:Task 1 started
INFO:__main__:Task 0 finished
INFO:__main__:Task 2 started
INFO:__main__:Task 1 finished
INFO:__main__:Task 3 started
INFO:__main__:Task 2 finished
INFO:__main__:Task 4 started
INFO:__main__:Task 3 finished
INFO:__main__:Task 4 finished


But I want logs that are not interleaved, and am willing to sacrifice real-time delivery of logs within each process to make this happen:

INFO:__main__:Task 1 started
INFO:__main__:Task 1 finished
INFO:__main__:Task 0 started
INFO:__main__:Task 0 finished
INFO:__main__:Task 4 started
INFO:__main__:Task 4 finished
INFO:__main__:Task 2 started
INFO:__main__:Task 2 finished
INFO:__main__:Task 3 started
INFO:__main__:Task 3 finished


Is there a straightforward way to get this type of behavior?

Continue reading...