[Python] python pickle size limit

i want to pickle a large (1810392*255) numpy array. However when pickling i get an error:

[...]error: 'i' format requires -2147483648 <= number <= 2147483647


Code:

import numpy
import pickle
l=numpy.zeros((1810392,255))
f=open('file.pkl','wb')
pickle.dump(l,f,2)


Is there a size limit? Is there a workaround? If not necessary I do not want to use hdf5 or something not build into python.

I also tried numpy.savez and numpy.savez_compressed. Code:

import numpy
l=numpy.zeros((1810392,255))
numpy.savez_compressed('file.npz',l)


Saving works but when i try to load the data I get an error. Code:

import numpy
l=numpy.load('file.npz')
l['arr_0']


I need to use numpy.savez instead of numpy.save because I want to store additional data.

Continue reading...