[Python] Ways to remotely debug Python scripts for model training on a separate servers while...

at my place of work we develop machine learning assisted process to help us with object detection in large scale remote sensing data. We have Server (running Windows) that houses the required software and GPUs for modeltraining. We would usually develop on this server using remote desktop connections. Now we have change in policy requiring us to run all user software with citrix. We are testing now a setup where one terminalserver houses all the required software (Visual Studio 2022, Anaconda etc) and the old development server has the resources for training and the files needed (the python files, the conda enviroments etc). I need to figure out how me and my collegues can write our code with Visual Studio over Citrix but use the resources of the the development server to run the code. Its important that we can also debug the code in Visual Studio while sill using the processing power of the other server. How can I develop code with Visual Studio 2022 running on Citrix while using the resources on another server? Is this even possible?

I can establish a ssh connection between the two servers and run the python files via commandline but this is not what we want. We want to efficienty debug Python code within an IDE. I also tried to get Nvidia Nsight for Visual Studio running but as it turned out that the included debugger for VS22 and Hopper GPUs doesnt support remote debugging at the moment. And Im also not sure if it would have worked with plain python scripts anyways. I want to test the remote development feature in VS Code but I first need to get this software approved which might not happen. I also looked into the paramiko package but as far as I understand it, the user has to include their user information with password in the code itself. Thats seems like a huge security risk.

Continue reading...