[Python] Switching to Python from MATLAB: plotting and RAM consumption

I'm in the process of switching from MATLAB to Python. I'm rewriting most of my codes and am using Spyder IDE.

I constantly need to load large datasets of data and plot it. Here I've encountered an issue with my use of Python and RAM consumption when plotting the data

Here's an example of two codes, first in MATLAB and the RAM usage at each point, and then implementation in Python. The goal is to read several HDF5 datasets, stitch them together and plot.

MATLAB:

filename = dir('*00.hdf5');
[~, idx] = max([filename.datenum]);
filename = filename(idx).name;
dLim = [0 1];
Data = [];
lgth = cellstr(num2str((dLim(1):dLim(2))', '%04d'));
date = filename(1:17);
for i = 1:length(lgth)
filename1 = [date ,sprintf('.00%s', lgth{i}),'.hdf5'];
intD = double(h5read(filename1,'/data'));
Data = [Data; intD];
end


Final size of Data = 58976577x4 RAM usage = 3662 MB

after plot with MATLAB

RAM usage = 7322 MB.

Python:

filenames = glob.glob('*00.hdf5')
filename = max(filenames, key=os.path.getmtime)
filename = os.path.basename(filename)
dLim = [0, 1]
lgth = [f"{i:04d}" for i in range(dLim[0], dLim[1] + 1)]
date = filename[:17]
Data = None
for i in range(len(dLim)):
filename1 = f"{date}.00{lgth}.hdf5"
with h5py.File(filename1, 'r') as f:
intD = np.array(f['/data']).T
if Data is None:
Data = intD
else:
Data = np.vstack((Data, intD))


Final size of Data = 58976577x4

RAM usage = 2280 MB

after plotting with matplotlib:

from matplotlib import pyplot as plt

plt.plot(Data)


RAM usage = 11000+ MB.

Summary: MATLAB RAM consumption:

Final size of Data = 58976577x4

RAM usage = 3662 MB

After plot:

RAM usage = 7322 MB

Python (Spyder) RAM consumption:

Final size of Data = 58976577x4

RAM usage = 2280 MB

After plot with matplotlib:

RAM usage = 11000+ MB. Becomes barely responsive.

What am I doing wrong? This is not even the largest dataset that I have worked with before, and I'm stuck... I can downsample the data and reduce the load, but that's not the point. That can be done in MATLAB as well...

Continue reading...