I am trying to setup an image pipeline: load images from disk into a dataset, rescale, generate patches, train. Loading from disk is slow so I want to create random data to simulate a real image. I have created a generator class which produces a random numpy array and converts it to a tensor then returns it. I then use Dataset.from_generator but I get an error when trying to iterate my data. This problem originally appeared when passing the dataset into model.fit but I have found I can trigger it sooner with the following import numpy as np import tensorflow as tf # batch size for dataset full_image_batch_size = 1 batch_size = 1 image_height = 3000 image_width = 5328 image_channels = 3 tile_size = 256 class FakeImageGenerator: def __init__(self, number_of_images): self.generated = 0 self.number_of_images = number_of_images def __len__(self): return self.number_of_images def __getitem__(self, index): arr = np.random.rand(image_height, image_width, image_channels) * 255.0 arr = tf.convert_to_tensor(arr, dtype=tf.float32) return (arr, ) def __call__(self): return self.__getitem__(0) x_image_generator = FakeImageGenerator(full_image_batch_size) x_train = tf.data.Dataset.from_generator( x_image_generator, output_signature=( tf.TensorSpec(shape=(image_height, image_width, image_channels), dtype=tf.float32), )) for elem in iter(x_train): print(elem) W tensorflow/core/framework/op_kernel.cc:1829] INVALID_ARGUMENT: TypeError: `generator` yielded an element that did not match the expected structure. The expected structure was (tf.float32,), but the yielded element was [[[ 59.90327 5.024378 118.99237 ]... [178.26884 200.46623 15.418176]]]. Traceback (most recent call last): File "/root/.virtualenvs/ga-python/lib/python3.12/site-packages/tensorflow/python/data/ops/from_generator_op.py", line 204, in generator_py_func flattened_values = nest.flatten_up_to(output_types, values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.virtualenvs/ga-python/lib/python3.12/site-packages/tensorflow/python/data/util/nest.py", line 237, in flatten_up_to return nest_util.flatten_up_to( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.virtualenvs/ga-python/lib/python3.12/site-packages/tensorflow/python/util/nest_util.py", line 1541, in flatten_up_to return _tf_data_flatten_up_to(shallow_tree, input_tree) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.virtualenvs/ga-python/lib/python3.12/site-packages/tensorflow/python/util/nest_util.py", line 1570, in _tf_data_flatten_up_to _tf_data_assert_shallow_structure(shallow_tree, input_tree) File "/root/.virtualenvs/ga-python/lib/python3.12/site-packages/tensorflow/python/util/nest_util.py", line 1414, in _tf_data_assert_shallow_structure raise TypeError( TypeError: If shallow structure is a sequence, input must also be a sequence. Input has type: 'EagerTensor'. Note: in the stack trace, the "yielded element was..." displays the entire numpy array but I have only included a couple of slices to be representative. Continue reading...
