[Python] Load class instance from pickle file during instantiation with HIGHEST_PROTOCOL

Goal

• When I create an instance of ClassA:
  • if a pickle file exists, then load the instance from pickle file
  • if a pickle file does not exist, then create the instance from scratch
• use pickle HIGHEST_PROTOCOL.

Fail


Someone reported here the same problem in a comment. The problem is also described in this post: if protocol > 1, then pickle calls __new__ during loading, which creates an infinite recursion.

# module_a.py
import os
import pickle
# import dill as pickle

save_path = r'C:\tests\pickle_tests\saved_instance_of_a.pkl'

def load(path):
with open(path, 'rb') as f:
return pickle.load(f)

def dump(x, path):
with open(path, 'wb') as f:
pickle.dump(
x, f,
protocol=pickle.HIGHEST_PROTOCOL)

class ClassA:
def __new__(cls):
print('__new__ called')

if os.path.isfile(save_path):
print('The saved pickle exists: loading from file.')
instance = load(save_path)

else:
print('The saved pickle does not exist: creating.')
instance = super(ClassA, cls).__new__(cls)

return instance

def __init__(self):
print('__init__ called')
if not os.path.isfile(save_path):
self.my_dict = {'pi': 3.14}
dump(self, save_path)


# myprogram.py
import os
import module_a

if __name__ =='__main__':
instance_a = module_a.ClassA()
print(instance_a.my_dict)


First run OK (creating instance from scratch):

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}



Second run fail (loading instance from pickle):

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
...
File "C:\tests\pickle_tests\module_a.py", line 19, in __new__
print('__new__ called')
File "C:\anaconda\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
RecursionError: maximum recursion depth exceeded while calling a Python object


Current workaround


The code above works if I replace protocol=pickle.HIGHEST_PROTOCOL by protocol=0 but I do not want to use protocol 0 (or 1). I want to use protocol=pickle.HIGHEST_PROTOCOL.

First run OK:

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}


Second run OK:

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__init__ called
{'pi': 3.14}

Continue reading...