[Python] Make Hydra to load configs from zip-archive

I am trying to adopt this wonderful repository with Lightning-Hydra template to my workflow.

My workflow looks like following.

1. 
   Create a new git branch for a new experiment.
   
   
2. 
   Write a code (plus configs) to implement an idea (including debugging and testing)
   
   
3. 
   Commit
   
   
4. 
   Pack everything into a zip archive (zip -rp experiment-$(git rev-parse HEAD).zip *)
   
   
5. 
   Add to the archive the file __main__.py, calling main()
   
   This file allows to call python experiment-aabbcc.zip
   
   Approximate content of this file is
   
   import sys
   from src.train import main
   main(sys.argv[1:])
6. 
   Submit to the job queue command python /path/to/experiment-aabbcc.zip --other --command --line-args.
   
   I use this fork of the task-spooler as a job scheduler. Because of technical limitations of a PC that I use, I cannot run docker (actually I ssh to a docker container that doesn't support docker-in-docker, and I cannot change that).
   
   
7. 
   git checkout master
   
   
8. 
   go to 1
   

The issue is that hydra + OmegaConf refuse to load configs from created zip archive, they always look into a file system.

Since I frequently switch branches, content of working directory often changes, and I have high chances that a job submitted to a job queue will run the code that is different from what was intended.

So, I am looking for a solution that would allow me to isolate a code, submitted to the task spooler, without making extra copies of a repo on a file system.

Continue reading...