[Python] AWS S3 ModuleNotFoundError: No module named Modulename

I'm trying to deploy a Python script on AWS Glue that depends on several custom Python modules. Despite following common practices, I'm consistently encountering a ModuleNotFoundError. I'm hoping someone can help me pinpoint what I might be missing. Problem: When my Glue job runs, I get the error: ModuleNotFoundError: No module named 'Modulename' (where Modulename is one of my custom modules like company_processors or check_works).

My Local Project Structure:

projectname/
├── app/
│ ├── core/
│ │ ├── cleansing.py # This is the main script uploaded to Glue
│ │ └── cleansingSubModules/
│ │ ├── __init__.py # (Empty file)
│ │ ├── company_processors.py
│ │ └── check_works.py
│ └── __init__.py # (Empty file, but not strictly part of the zip's core dependency)
└── (other project files)


Import Statements in cleansing.py (The script uploaded to Glue):

My cleansing.py script, located at projectname/app/core/cleansing.py, contains the following exact import statements for my custom modules:

# Other standard imports...
import boto3
# ...

# Custom module imports:
from check_works import print_hello
from company_processors import process_employee_sizes, process_industry, process_revenue


# ... rest of the script (Cleaning class, main execution)


How I Created the Zip File for AWS Glue:

To create the cleansing_modules_for_glue.zip file, I performed the following steps on my Windows system:

1. Navigated into the cleansingSubModules directory: C:\path\to\projectname\app\core\cleansingSubModules\
2. Ensured __init__.py was present and empty in this directory.
3. Selected ALL the contents of cleansingSubModules (i.e., __init__.py, company_processors.py, contact_processors.py, etc.).
4. Right-clicked on the selected files and chose "Send to" -> "Compressed (zipped) folder".
5. Renamed the resulting zip file to cleansing_modules_for_glue.zip.

Verified Content of cleansing_modules_for_glue.zip (by opening it):

When I open cleansing_modules_for_glue.zip, I see the following structure directly at its root:

cleansing_modules_for_glue.zip
├── __init__.py
├── company_processors.py
├── contact_processors.py
├── job_processors.py
├── address_processors.py
├── mobile_processor.py
└── check_works.py


There are no intermediate folders like app/ or cleansingSubModules/ inside the zip's top level.

AWS Glue Job Configuration:

1. cleansing.py is uploaded to s3://bcuket/glue-scripts/cleansing.py.
2. cleansing_modules_for_glue.zip is uploaded to s3://bcuket/cleansing_glue_libs/cleansing_modules_for_glue.zip.
3. In the Glue Job Details:
   • Python script path: s3://bucket1/glue-scripts/cleansing.py
   • Python library path: s3://bucket2/cleansing_glue_libs/cleansing_modules_for_glue.zip

Debugging Steps Already Taken:

1. Confirmed Zip Content: As described above, I have opened the zip file and verified that all .py files (including __init__.py) are directly at its root.
2. Confirmed Import Statements: The imports in cleansing.py (e.g., from company_processors import ...) directly match the module names at the root of the zip.
3. Case Sensitivity: Verified that the casing of filenames in the zip matches the casing in the import statements (e.g., company_processors.py vs. from company_processors).

Despite these checks, the ModuleNotFoundError persists. I'm running out of ideas. Any insights or suggestions would be greatly appreciated!

Continue reading...