[Python] Why is AWS SageMaker not submitting a training job when I create an estimator object?

I'm trying to submit a training job to AWS SageMaker AI API via GitHub using the HuggingFace estimator. However, every time I try to create the job, the estimator creates an object of type None and the script hangs up indefinitely. It never finishes executing and a training job is never even submitted to AWS SageMaker, as both the UI and the CLI show no training jobs when I check them.

Here is the code I'm using, with sensitive information removed:

# Import modules
import sagemaker
from sagemaker.huggingface import HuggingFace
import logging

# Troubleshoot errors
logging.getLogger("sagemaker").setLevel(logging.DEBUG)

# Create session
session = sagemaker.Session()

# Config AWS role
role = [AWS ARN]

# Config various login info
git_config = {
[REPOSITORY LOGIN INFO]
}
hf_token = [HUGGINGFACE LOGIN INFO]

# Configure training container
print("Creating estimator object...")
estimator = HuggingFace(
role=role,
session=session,
instance_count=1,
source_dir="jobs", # Subdirectory in repo
py_version='py310',
git_config=git_config,
entry_point="train.py", # Name of training script in GitHub
requirements_file="requirements.txt", # Additional packages to install
output_path='s3://llama-data/model-output/', # Output of model checkpoints
instance_type="ml.p3.2xlarge",
environment={
"HF_TOKEN": hf_token
},
enable_sagemaker_metrics=True, # Log in CloudWatch
image_uri='763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:2.1.0-transformers4.36.0-gpu-py310-cu121-ubuntu20.04', # Image to pull
hyperparameters=None, # All hyperparams declared in train.py
training_repository_access_mode='Platform' # Docker image is in Amazon ECR
)
print("Estimator object created successfully")

# Configure training/validation data
train_input = sagemaker.inputs.TrainingInput(
s3_data="s3://llama-data/train/",
content_type="text/csv"
)

validation_input = sagemaker.inputs.TrainingInput(
s3_data="s3://llama-data/validate/",
content_type="text/csv"
)

# Debugging: Print the SageMaker training request before submission
# request_dict = estimator.latest_training_job.describe_request()
print("SageMaker Training Job Request:", estimator.latest_training_job) # **THIS ALWAYS PRINTS 'NONE', NO REQUEST IS SUBMITTED**

print("Calling .fit()...")
estimator.fit({ # **SCRIPT STOPS WORKING AROUND THIS POINT**
"train": train_input,
"validate": validation_input
}, wait=True, job_name="train-llama-small", logs="All")
print("Training job created successfully. Exiting script.")


After the estimator is created I try to list the most recent training job, and this is printed:

Estimator object created successfully
SageMaker Training Job Request: None


When the script hangs up, this is the message I see in my terminal:

Your branch is up to date with 'origin/main'.
[03/07/25 09:31:21] DEBUG Using provided s3_resource fw_utils.py:474
[03/07/25 09:31:22] DEBUG Train args after processing estimator.py:2513
defaults:


And then it lists all of the training arguments for the session. I've tried pulling a different Docker image to see if that would change anything, with no effect.

What am I doing wrong that the training job is never actually created?

Continue reading...