I'm working on a CNN project where I make use of TensorFlow and Keras. In addition, I use interleave as the dataset is rather large (too large to load into the RAM with our resources). But there's an issue somewhere as the training gets stuck at a very low accuracy and is very jumpy. In addition, I get the warning: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches. You may need to use the .repeat() function when building your dataset. The dataset is made up of TXT files holding 3D arrays of floats where the 2 inner dimensions are attribute sets, and the outer the amount of attribute sets (which may vary between files). Meaning that each TXT files holds multiple entries to be used for training. These files are then stored in labeled folders.At the moment, the preparation of the dataset looks something like what's seen below. As the TXT files holds different amounts of entries, I try to write the pathToDataset() function to generate Datasets that is able to parse single entries and not batches at a time. # Define the processTXTFile_tf function def processTXTFile_tf(file_path): # As the file_path is a tensor, we need to decode it file_path = file_path.numpy().decode('utf-8') # Get MFCC NP array mfccArray = readTXTFile(file_path) # Add color channel mfccArray = np.expand_dims(mfccArray, axis=-1) # Convert to tensor tensor = tf.convert_to_tensor(mfccArray, dtype=tf.float16) #print(f"tensor shape: {tensor.shape}") # Return the tensor return tensor # Create a dataset from a file path def pathToDataset(file_paths): return ( tf.data.Dataset.from_tensor_slices(file_paths) .interleave( lambda file: tf.data.Dataset.from_tensor_slices( tf.py_function(processTXTFile_tf, [file], [tf.float16]) ), #cycle_length=min(train_len, 16), num_parallel_calls=tf.data.AUTOTUNE, ) .unbatch() .map(lambda x: tf.ensure_shape(x, INPUT_SHAPE)) # Ensure correct shape ) # Create the x_train_tf dataset x_train_tf = pathToDataset(x_train) # Create the y_train_tf dataset y_train_tf = tf.data.Dataset.from_tensor_slices(y_train) # Combine x_train_tf and y_train_tf into a single dataset train_tf = tf.data.Dataset.zip((x_train_tf, y_train_tf)).shuffle(buffer_size=len(x_train), reshuffle_each_iteration=True).batch(BATCH_SIZE).repeat().prefetch(tf.data.AUTOTUNE) # Removed shuffle(buffer_size=len(x_train)) from after batch() # Create the x_val_tf dataset x_val_tf = pathToDataset(x_val) # Create the y_val_tf dataset y_val_tf = tf.data.Dataset.from_tensor_slices(y_val) # Combine x_val_tf and y_val_tf into a single dataset val_tf = tf.data.Dataset.zip((x_val_tf, y_val_tf)).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) # Removed shuffle(buffer_size=len(x_train)) from after batch() The full code is available here: Google Colab And the current data set is here: ZIP on google Drive I believe the issue is related to the warning, and that the interleaves/database object aren't working as supposed to... But what exactly is the issue and how to resolve it? Continue reading...
