I'm running physical simulations and storing the data for later analysis. Up to this point, I've been storing the data as a pickled data frame, and manipulating it using pandas in jupyter lab. However, as my simulations are getting larger, unpickling the entire dataset is making the notebook run very slowly. Often, I only need a subset of the data - a few columns. I've resorted to saving some of the datasets as csv's, since I can import specific columns from that format. The full datasets can be in the 1-2 GB range - potentially larger in the future, but that's the extent right now. Is there are format, short of a database, that can allow quick access to a specified subset of a large data frame? Or is a database the way to go? Continue reading...
