[Python] Incorrect header check error when trying to access a tf_record file

I've recently ran into an error when trying to evaluate my training model onto my testing files. The error is thrown at the batch_raw = raw_iter.get_next():

try:
while N_b<max_batches:
batch_raw = raw_iter.get_next()
batch_test = test_iter.get_next()
if N_b%20==0:
tqdm.write('Processing batch '+str(N_b))

N_b = self.predict_batch(model, batch_test, batch_raw, csvPath, N_b, vars_to_save, plotter)
except tf.errors.OutOfRangeError:
pass


The error I'm getting is the following:

Traceback (most recent call last):
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 2113, in execution_mode
yield
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 730, in _next_internal
ret = gen_dataset_ops.iterator_get_next(
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2579, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6862, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.DataLossError: inflate() failed with error -3: incorrect header check [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/plebani/met-ml/met-ml/share/example_METNetRunner.py", line 391, in <module>
run('METNetSig', loss_fn=loss_fn, weighting=weighting, rotation=True, class_sig=False, metnetsig=False)
File "/home/plebani/met-ml/met-ml/share/example_METNetRunner.py", line 355, in run
x.get_predictions(x.outDir + 'model', plot=True, sig_plots=True, saveSUSY2CSV=True)
File "/home/plebani/met-ml/met-ml/src/METNetRunner.py", line 517, in get_predictions
self.predict_topology(model,top_name, test_dat, raw_dat, plot=plot, sig_plots=sig_plots, saveSUSY2CSV=saveSUSY2CSV, vars_to_save=self.vars_to_save, max_batches=100)
File "/home/plebani/met-ml/met-ml/src/METNetRunner.py", line 358, in predict_topology
batch_raw = raw_iter.get_next()
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 800, in get_next
return self._next_internal()
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 739, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 2116, in execution_mode
executor_new.wait()
File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/executor.py", line 69, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.DataLossError: inflate() failed with error -3: incorrect header check


The error suggests that the files are corrupted, but I made multiple tests that prove the opposite:

1. Try to inspect the file. For this I used the following snippet, I get as output the dataset containing all the information inside

filename = 'Zee_0__test1.tfrecord'
dataset = tf.data.TFRecordDataset(filename, compression_type='GZIP') # the files are compressed
for record in dataset.take(1):
print(record)

1. 
   Run the code again -> sometimes the code doesn't give this error and it's able to run properly, whereas other times (most of them) the error appears again.
   
   
2. 
   If I run in batches (meaning setting max_batches > 1, does 10k events at a time out of a total of 80k events), I get the error, whereas if I'm running only for one batch (10k events) I don't get the error. But if I count the number of batches processes before it gives the error, it's different every time.
   

I am using a comda environment with tensorflow 2.4.1. I tried to recreate the environment both with conda and with a python venv, and I also tried to update tensorflow to the latest version, but I still get the same error. I also tried to recreate the training files, but got the same outcome.

Is anyone able to help me on this? I'm running out of ideas

Continue reading...