[Python] Why is training multiple deep learning models (via multiple terminals) faster than...

I'm training deep learning models using TensorFlow (with GPU support) on my local machine. I noticed a surprising behavior:

• 
  When I train just one model (in a single terminal), it runs slower.
  
  
• 
  But when I open multiple terminals and run several training scripts at the same time, not only does the total training time decrease, but each model also trains faster than when run alone.
  

GPU: NVIDIA RTX 4070

Framework: TensorFlow 2.10.1

OS: Windows 11

Each terminal runs a separate Python process training the same model on the same data and seed.

Here is my part of python script:

gpus = tf.config.list_physical_devices('GPU')
if gpus:
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)

grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1, cv=3)
start_time = time.time()
grid_result = grid.fit(X, Y)
end_time = time.time()


My guess is: when only one model is running, it utilizes fewer GPU cores; but when multiple models run in parallel, the GPU becomes more fully utilized, and even a single model may end up using more GPU cores due to better overall scheduling.

Continue reading...