[Python] Feedback needed on building a Python DAG with caching and disk persistence and...

I am looking for some feedback on what I am doing and on my plan on building automatic dependency graph functionality by inspecting the call stack. The background is below.

I am working on an academic project and I am developing a number of mathematical models. I have structured my code as a DAG (directed acyclic graph). This is quite natural and in Python each of the models look like a property which return a pandas DataFrame. For example, if I have modelA and modelB where model B uses model A as one of its inputs this would look like (ignore modelC for now):

class Models:
@cached_property
def modelA(self):
result = ...
return result

@cached_property
def modelB(self):
result = funcB(self.modelA, other_inputsB)
return result

@cached_property
def modelC(self):
result = funcC(self.modelB, other_inputsC)
return result


Every model is also marked with @cached_property. This is a rather simple wrapper which is similar to functools.cached_property but in addition it also checks if the result is persisted on disk and if it is not, it persists it to disk after calculation. It also exposes the in-memory cache as a dictionary so that it can be invalidated easily.

This works great so far. However, a key issue here is that if the calculation is changed, the onus is on the user to remember to invalidate the on-disk cache and to do this efficiently (e.g. delete only the persisted results for the modified models rather than everything).

A better solution would be for the caching system to detect that the model has changed. A way this can be achieved is to include the source code of the model in the cache hash key which in turn can be achieved using inspect.getsource(). This will solve the problem that if modelB is changed, the results persisted in the cache for modelB will become invalid (because the key will change so they can't be found), this doesn't solve the problem with modelC which depends on modelB and also needs to be recalculated.

To solve this, a dependency graph needs to be built so that modelC can know that it depends on modelB which has changed. When the cache is completely wiped out and everything needs to be recalculated from scratch, the system can build a dependency graph by inspecting the call stack in @cached_property. This can be done using traceback.extract_stack() module in python and inspecting the call stack.

Other frameworks that work to solve similar problems are listed below. I initially considered using them because they offer a lot of functionality. However, as far as I can tell in each, the user must explicitly define the call graph. They are also far more complicated where as all I needed and have built is a very thin wrapper that knows how to persist to and load from disk and exposes the cache as a dict. The functionality that is missing is the automatic dependency build and adding the source code as part of the hash key.

1. hamilton
2. luigi
3. dask
4. prefect
5. airflow
6. dagster

Continue reading...