[Python] Tensorflow dataset running out of data durin training

I'm creating a dataset from a file containing path-names to image locations and mask-locations (xtr and ytr). Initially, the csv contains approx. 1000 elements. Each image is than split using flat_map into 4 or 8 images depending on the height of the image. So, the dataset should still have at least 1000 elements.

Here is my pipeline:

train_ds = (
tf.data.Dataset.from_tensor_slices((xtr, ytr))
.shuffle(200)
.map(self._load_image, num_parallel_calls=tf.data.AUTOTUNE)
.map(self._preprocess, num_parallel_calls=tf.data.AUTOTUNE)
.flat_map(self._split)
.map(self._augment)
.batch(self._batch_size, drop_remainder=True)
.prefetch(tf.data.experimental.AUTOTUNE)
)


Counting images in the dataset usually returns about 1700 elements (after splitting etc.) Training is done with a batch-size of 4 and approx. 250 steps per epoch.

My problem is that I receive errors during training indicating that I run out of data. E.g. Local rendezvous is aborting with status: OUT_OF_RANGE

Sometimes I also get explicite UserWarnings about running out of data.

Any pointers on how to solve this?

Thanks

Continue reading...