I am working on a project which includes building a database based on a Simulink model with a lot of varying parameters. This model is simulated many times (over 100), and each run generates lots of time series, which I then later use for training and testing some regression models. At first it was purely a MATLAB project, and since the goal was just to test some ideas I didn't give much thought at the time to how the data was saved. Now, however, the project has become bigger, and some Python-programming colleagues have joined to work in the regression models. Since I have only dabbled in working with large datasets in Python, I would like some help on how to make my pipeline more efficient and a little more future-proof. The simulation, saving and training process currently looks like this: MATLAB: Initialize parameters for a batch of simulations. Run simulations in parallel (parsim), logging the simulation timeseries as timetables in a .mat file. Each file has around 500 MB, and contains two structures: SimulationMetadata, with execution info, and logout, with the timetables themselves. This format seems kinda fixed, as it is the standard way of logging simulations in parsim. All variables are sampled at the same rate and for the same duration. After running the simulations, read each .mat file, convert timetables to arrays and then overwrite the original .mat file. After that, the data is used for training regression models in either MATLAB as a FileDatastore; or in Python using scipy.io.loadmat. The main requisites for this project are as follows: Data must be readable in Python and MATLAB The dataset must be scalable, as I will probably need to run more simulations in the future I am not used to dealing with large databases in Python, so I'm open to any ideas you guys might have. Continue reading...
