[Python] How Does Airflow Handle Multiprocessing Tasks on an 8-Core CPU When parallelism Is...

I’m using Apache Airflow with the LocalExecutor on a machine that has an 8-core CPU.

Let’s say I have a DAG with two tasks that can run in parallel, and each of these tasks uses Python's multiprocessing module to spawn 5 subprocesses internally.

This leads me to some confusion:

That’s effectively 2 tasks × 5 processes = 10 total subprocesses, all trying to run on an 8-core machine. I’ve seen that in the default airflow.cfg, the parallelism value is set to 32. My questions are:

How does Airflow handle this scenario?

If each task internally spawns multiple subprocesses, won’t this lead to more processes than available CPU cores?

Is Airflow aware of the internal multiprocessing done inside the tasks?

Will it take into account those internal subprocesses when managing task concurrency?

Why is parallelism set to 32 by default in airflow.cfg?

On an 8-core machine, wouldn’t that allow way more tasks than my CPU can handle efficiently?.Say I have 32 task to run in parallel in airflow but I have 8 core cpu how is that even possible ?

I'm trying to understand how Airflow balances these configurations with the actual hardware resources available, and how I should approach setting these values in a real-world scenario.

Any clarification or insights would be greatly appreciated. Thanks!

Continue reading...