I'm trying to load a model that I have trained in Google Colab, using Ubuntu 24.04 with conda 24.5.0, NVIDIA-SMI 555.58.02, Driver Version 556.12, CUDA 12.5 All my drivers are updated, but when I try to run the follwing code, it gets stucked loading the model. I have already tried to import the Model function from tensorflow.keras.models and load directly from keras, with the same results. I had to downgrade python from 3.12 to 3.9, otherwise tensorflow wouldn't work, I also have tried to load other models that are loaded in colab with no problem, with similar results. I have the option to save the model as .keras, but when I do, it gives me Error loading model: SavedModel file does not exist at: model4.keras/{saved_model.pbtxt|saved_model.pb}. Should I try a specific combination of package version for this code to work? Any direction would be welcome. import tensorflow as tf from tensorflow import keras print(tf.test.is_built_with_cuda()) print(tf.__version__) try: model1 = keras.models.load_model('model.h5') print("Model loaded successfully") except OSError as e: print(f"Error loading model: {e}") exit() This is the full output: 2024-07-12 15:01:09.226398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 True 2.4.1 2024-07-12 15:01:09.624582: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2024-07-12 15:01:09.625313: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2024-07-12 15:01:09.728371: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.728405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 4070 computeCapability: 8.9 coreClock: 2.52GHz coreCount: 46 deviceMemorySize: 11.99GiB deviceMemoryBandwidth: 469.43GiB/s 2024-07-12 15:01:09.728420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2024-07-12 15:01:09.729041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2024-07-12 15:01:09.729073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2024-07-12 15:01:09.729788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2024-07-12 15:01:09.729909: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2024-07-12 15:01:09.730543: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2024-07-12 15:01:09.730889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2024-07-12 15:01:09.732249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2024-07-12 15:01:09.732301: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.732325: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.732338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2024-07-12 15:01:09.732474: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-07-12 15:01:09.734681: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2024-07-12 15:01:09.734728: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.734740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 4070 computeCapability: 8.9 coreClock: 2.52GHz coreCount: 46 deviceMemorySize: 11.99GiB deviceMemoryBandwidth: 469.43GiB/s 2024-07-12 15:01:09.734745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2024-07-12 15:01:09.734750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2024-07-12 15:01:09.734754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2024-07-12 15:01:09.734757: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2024-07-12 15:01:09.734760: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2024-07-12 15:01:09.734763: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2024-07-12 15:01:09.734780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2024-07-12 15:01:09.734796: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2024-07-12 15:01:09.734809: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.734818: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:01:09.734821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2024-07-12 15:01:09.734835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2024-07-12 15:02:14.023639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2024-07-12 15:02:14.023660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2024-07-12 15:02:14.023672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2024-07-12 15:02:14.024163: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:02:14.024181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1489] Could not identify NUMA node of platform GPU id 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2024-07-12 15:02:14.024211: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:02:14.024234: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:927] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-07-12 15:02:14.024260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10293 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 4070, pci bus id: 0000:01:00.0, compute capability: 8.9) Traceback (most recent call last): File "/home/abner/density/teste.py", line 8, in <module> model1 = keras.models.load_model('model.h5') File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/saving/save.py", line 206, in load_model return hdf5_format.load_model_from_hdf5(filepath, custom_objects, File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 183, in load_model_from_hdf5 model = model_config_lib.model_from_config(model_config, File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/saving/model_config.py", line 64, in model_from_config return deserialize(config, custom_objects=custom_objects) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize return generic_utils.deserialize_keras_object( File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object return cls.from_config( File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config input_tensors, output_tensors, created_layers = reconstruct_from_config( File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config process_layer(layer_data) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer layer = deserialize_layer(layer_data, custom_objects=custom_objects) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize return generic_utils.deserialize_keras_object( File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 360, in deserialize_keras_object return cls.from_config(cls_config) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 720, in from_config return cls(**config) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/layers/pooling.py", line 862, in __init__ super(GlobalPooling2D, self).__init__(**kwargs) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/training/tracking/base.py", line 517, in _method_wrapper result = method(self, *args, **kwargs) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 340, in __init__ generic_utils.validate_kwargs(kwargs, allowed_kwargs) File "/home/abner/anaconda3/envs/density/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 808, in validate_kwargs raise TypeError(error_message, kwarg) TypeError: ('Keyword argument not understood:', 'keepdims') Continue reading...
