[Python] TensorFlow does not accept list type for dataset generator

I am building a neural network. I couldn't load all the training data into memory at once, so I am using TensorFlow's tf.data.Dataset.from_generator function to load data incrementally. However, it throws an error saying it does not accept a list of tensors as a type.

TypeError: output_signature must contain objects that are subclass of tf.TypeSpec but found <class 'list'> which is not.

The input to my neural network is a list of 151 separate tensors. How can I represent this in the generator? My code is below:

def generator(file_paths, batch_size, files_per_batch, tam, value):
return tf.data.Dataset.from_generator(
lambda: data_generator(file_paths, batch_size, files_per_batch, tam, value),
output_signature=(
[tf.TensorSpec(shape=(batch_size, tam), dtype=tf.float32) for _ in range(tam+1)], # Lista de 151 tensores
tf.TensorSpec(shape=(batch_size, tam), dtype=tf.float32) # Rótulos
)
)

inputArray = [Input(shape=(tam,)) for _ in range(tam + 1)]

train_dataset = generator(file_paths, batch_size, files_per_batch, tam, False)
train_dataset = train_dataset.prefetch(tf.data.AUTOTUNE)

model.fit(train_dataset, epochs=1000, validation_split=0.2, verbose=1)


I tried to use tf.data.Dataset.from_generator to feed data into my neural network in batches, since I can't load all the data into memory at once. However, I encountered an error: TypeError: output_signature must contain objects that are subclass of tf.TypeSpec but found <class 'list'> which is not.

Continue reading...