I am a google colab pro+ user. I am trying to train a DL model and getting an error of "Transport endpoint is not connected", while training the model. I ve been using the pro version previously and faced the same error so I thought of upgrading but it didnt help as well. The code was running fine for 3hrs and everytime it is around 3-4hrs, I face the error. I tried to keep the runtime continue using a javascript and it was working fine. The full error info is: Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 18% 54466/300000 [3:57:19<17:49:51, 3.83it/s] Traceback (most recent call last): File "/content/drive/MyDrive/Ub4D/exp_runner.py", line 998, in <module> File "/content/drive/MyDrive/Ub4D/exp_runner.py", line 257, in train File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 391, in add_scalar self._get_file_writer().add_summary(summary, global_step, walltime) File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 113, in add_summary self.run() File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 244, in run self.add_event(event, global_step, walltime) File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 98, in add_event self._run() self.event_writer.add_event(event) File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 117, in add_event File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 275, in _run self._record_writer.write(data) File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/record_writer.py", line 40, in write self._async_writer.write(event.SerializeToString()) File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 171, in write self._writer.write(header + header_crc + data + footer_crc) File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/lib/io/file_io.py", line 101, in write self._check_worker_status() File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 212, in _check_worker_status self._writable_file.append( raise exception tensorflow.python.framework.errors_impl.FailedPreconditionError: exp/Cactus/paper_config/logs/events.out.tfevents.1691166619.f28574666dfe.12707.0; Transport endpoint is not connected File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 244, in run self._run() File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 275, in _run self._record_writer.write(data) File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/record_writer.py", line 40, in write self._writer.write(header + header_crc + data + footer_crc) File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/lib/io/file_io.py", line 101, in write self._writable_file.append( tensorflow.python.framework.errors_impl.FailedPreconditionError: exp/Cactus/paper_config/logs/events.out.tfevents.1691166619.f28574666dfe.12707.0; Transport endpoint is not connected Continue reading...
