[Python] how to handle caching with big data in python

I have model composed of multiple components. I have created solver where it take a "state" of each component (which is described as dictionary of {component: state}) and provide me value for each component into details as a dictionary as well ({component1: {key1: value1, key2: value2}, component2: {keyx: valueX, key3: valueyY}).

Model components number can be thousands. In addition, solving function is wasting a lot of time, for that, since each "state" can be repeated, I have created a cache where key is the state and value is the output of the solve. Hence, I won't recalculate if same state twice.

Leveraging that, I have also created a cache file, where I can reuse same cache next time and no need to recalculate.

in terms of performance, I have save a lot of time!!

In the second hand, I have created a memory issue

Here is the flow:

1. running the program
2. converting "state" to hash using hashlib function
3. saving hash -> results ({componnet1: {key: val}..., ...}) in data_cache dict
4. everytime i call solve, i convert state to hash, check if it has in data_cache dict 4.1 if exist, return results 4.2 solve & save
5. once done, I save the data_cache to json compressed with lz4 library

in the next run, if cache file exist then i load it before start solving.

Issues:

1. cache is getting soo big in process ~10GB (my model can be thousands of component and each component might have thousands in resutls)
2. Time paying in loading and caching (one time, I can live with that)
3. cache file is getting big the more combinations I have

how to handle such situation where I still benefit from performance boost yet dont create memory issue in the program

1. creating memory issue

Continue reading...