[Python] Kernel dying while training a H-MRCNN model

I am training an MRCNN model on a hyperspectral dataset of 102 images (.tif) and labels (.mat).

Specifications:

GPU: RTX 3060
Python: 3.9.21
Tensorflow: 2.10.0
Numpy: 1.23.5
CUDA: 11.5
Keras: 2.10.0


Sample input shape: (1, 64, 64, 4)
Sample label shape: (1, 64, 64, 4)
Unique label values:
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 56 58 59 60 61 62 63 67 70 74 255]


CUDA uses only 6GB of GPU.

I can't change any of the specifications. I need the kernel to stop dying and my model to train without flaw.

Please help ASAP.

Continue reading...