[Python] TensorFlow hangs indefinitely on model.fit() even with synthetic data

I have my dataset with images to classify, using an ML model in TensorFlow. When I train the model for a number of epochs, TensorFlow automatically prints out which epoch we are on. Mine is always stuck on 1/n. None of the batches are ever processed.

In Jupyter notebooks the process can run indefinitely, I left it for an hour and it was still on 1 epoch out of 2. I checked in Task Manager whether GPU is utilized while the thing hangs, and it wasn't - 0% usage. The size of dataset was tiny - just couple thousand low-res images. In raw python file, however, after some time the process breaks.

Specs:

• TensorFlow: tensorflow-gpu 2.9.1
• Windows 10
• Python: 3.9.13
• IDE: PyCharm
• cuDNN: 8.1
• CUDA: 11.2
• CPU: Ryzen 5 5600
• GPU: NVIDIA RTX 3070Ti

Epoch 1/2

Process finished with exit code -1073740791 (0xC0000409)

I read about it and its something about memory buffer overflow or something like that, so what I did was creating a small synthetic dataset to test it out - just 10 random 20x20 images:

model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(20, 20, 3)),
tf.keras.layers.Conv2D(16, 3, activation='relu'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy')
model.summary()

x = tf.random.normal((10, 20, 20, 3))
y = tf.random.uniform((10,), maxval=2, dtype=tf.int32)

model.fit(x, y, epochs=2)


I still get the same result:

Epoch 1/2

Process finished with exit code -1073740791 (0xC0000409)


I even tried writing a custom logger to log end of each batch so I could see that at least something was calculated:

class BatchLogger(tf.keras.callbacks.Callback):
def on_train_batch_start(self, batch, logs=None):
print(f"▶Starting batch {batch}")

def on_train_batch_end(self, batch, logs=None):
print(f"Finished batch {batch} | loss: {logs['loss']:.4f}, acc: {logs['accuracy']:.4f}")


And pass it through callbacks:

history = model.fit(
train_ds,
validation_data=val_ds,
epochs=10,
callbacks=[checkpoint_cb, earlystop_cb, tensorboard_cb, BatchLogger()]
)


But still with no result. I tested whether GPU is detected:

print("TensorFlow version:", tf.__version__)
gpus = tf.config.list_physical_devices('GPU')
print("Num GPUs Available:", len(gpus))

for i, gpu in enumerate(gpus):
details = tf.config.experimental.get_device_details(gpu)
print(f"\nGPU {i}:")
for k, v in details.items():
print(f" {k}: {v}")


And indeed it is detected:

TensorFlow version: 2.9.1
Num GPUs Available: 1

GPU 0:
device_name: NVIDIA GeForce RTX 3070 Ti
compute_capability: (8, 6)


Later I decided to test whether GPU is actually used by Tensorflow - I measured time needed for my CPU and GPU to multiply a 16384 x 16384 matrix by itself.

# Force CPU
with tf.device('/CPU:0'):
print("\n Running on CPU...")
start = time.time()
c_cpu = tf.matmul(a, b)
tf.experimental.numpy.sum(c_cpu) # force evaluation
end = time.time()
print(f"CPU Time: {end - start:.4f} seconds")

# Force GPU
if tf.config.list_physical_devices('GPU'):
with tf.device('/GPU:0'):
print("\n Running on GPU...")
start = time.time()
c_gpu = tf.matmul(a, b)
tf.experimental.numpy.sum(c_gpu) # force evaluation
end = time.time()
print(f"GPU Time: {end - start:.4f} seconds")
else:
print("\nNo GPU available.")



The GPU was indeed utilized. I saw a 20x difference between these.

CPU Time: 16.1876 seconds
GPU Time: 0.8211 seconds


So its not a hardware problem or cuDNN/CUDA problem I believe. I started messing with Python versions and Tensorflow versions, here is what I tried:

• Using tensorflow 2.10.0
• Using tensorflow-gpu 2.10.0
• Using tensorflow-gpu 2.9.1
• Using python 3.10

Pretty much it. No clue outside of that, every version of TensorFlow had exactly the same issue for me.

If somebody had this issue before I'd be grateful for some insights. Thanks!

Continue reading...