[Python] Best-practices for maintaining the provenance of the output of an evolving codebase

I maintain a python simulator which ingests a python configuration object, defining the parameters of the simulation, and emits many small objects the simulation results. Both the input configuration objects and the codebase change frequently. Also changing frequently are third-party dependencies.

Now suppose a future me has a question about a simulation result from a year ago. I have two problems.

1. How do I automatically record the provenance of simulation results, i.e. which versions of all code-bases were used to generate a given result, so that I can later reproduce/debug the simulation result in question?
2. And I want to do this in such a way that it is relatively painless to revert the code-base (and all third-party dependencies) to the versions.

I already use conda, git, etc. My current workaround for the first problem is to import all third-party packages and log their respective git-hashes every time the codebase runs. I don't have a good workflow for the second other than building another environment, which is slow.

I'm looking for best-practices tips and to learn whether other folks encounter similar problems and how they address it.

Continue reading...