[Python] Best way to create a Python virtual environment with a higher Python version than the...

I work on a computing cluster. The global Python version on the node is 3.8. I have a virtual environment that I use when I start a job. The cluster uses LSF for scheduling. I usually access it via Open On Demand, start a Jupyter notebook, and switch the kernel to the virtual environment the contains all the dependencies of my project.

Now I want to use a Python library that requires Python version >=3.9. If I was just using bsub to submit jobs I would create a docker image with everything I need in it. But I would prefer to work in a Jupyter notebook. It appears that I cannot create a virtual environment with a Python version higher than the global version.

I have contacted support but let's assume I cannot change the global python version on the node. What is the best way to make my project work if some dependencies require Python 3.9? I have thought about trying to use a Docker image to create a virtual environment on the disk, but I am not sure if that will work.

I would appreciate any help in this matter.

Continue reading...