[Python] Ray parallelised function intermittently cannot find package

I have the following directory structure for my project and a local repository of utility functions:

mydir/
- proj/
- main_script.py
- utils/
- package1/
- __init__.py
- module1.py
- module2.py
- package2/


My main script contains a Ray parallelised function that calls a function from the utilities directory. I perform the import and Ray initialisation as follows:

import os
import sys
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, BASE_DIR)
from utils.package1 import module1

@ray.remote
def fcn_parallel(input_args):
*some code here*
module1.do_something()
*some code here*

ray.init(runtime_env={'py_modules': [module1]}, num_cpus=4)
obj_refs = [fcn_parallel.remote(input_args) for i in range(n)]
ray.get(obj_refs)


This worked for a while, but I am now getting a Module not found error when getting the object refs.

I cannot pinpoint when exactly it stopped working because I cannot reproduce the error consistently. Sometimes not importing an unrelated package works once, but when I restart the kernel and rerun I get the same error. The only fix I have found so far is to add the following:

os.environ['PYTHONPATH'] = 'C:/some_path/mydir/'
os.environ['RAY_RUNTIME_ENV_WORKING_DIR_CACHE_SIZE_GB'] = '0'


which seems strange since I have already added BASE_DIR to the path. BASE_DIR is the same as C:/some_path/mydir/. If I do not parallelise with Ray and simply run in serial, all packages and modules can be found and imported just fine without having to add those two lines.

I found the fix above from: ModuleNotFoundError seen after the first time a job is run on a Ray cluster. The accepted answer also says to add the runtime_env key during ray.init(), which I did but did not solve the issue.

Can someone explain why this is happening but also why it has been working fine for nearly a year before intermittently not working?

Continue reading...