[Python] Can't get hyperparameters to pass in as arguments to a sagemaker estimator when using...

I just need to pass hyperparameters to my training job when using a custom docker image

When I remove the custom image "image_uri" I can see the arguments being passed, but things don't work as I need the custom image.

I also tried using just a regular sagemaker

Estimator(
image_uri=image_uri,
role=role,
instance_count=1,
instance_type="ml.m5.large",
hyperparameters=hyperparameters,
output_path=f"s3://{ML_S3_BUCKET}/models/{training_job_name}",
)


as well

Starting the training job

hyperparameters={
"epochs": 3000,
"learning_rate": 0.0005,
"input_size": 57,
"hidden_size": 40,
"output_size": 1,
"patience": 1000,
"hidden_layer_count": 4,
"is_alpha": False,
}
estimator = PyTorch(
entry_point="train/train_illuvitar_neural_net.py",
image_uri=image_uri,
role=role,
py_version="py310",
framework_version="2.0",
instance_count=1,
instance_type="ml.m5.large",
hyperparameters=hyperparameters,
output_path=f"s3://{ML_S3_BUCKET}/models/{training_job_name}",
)


Dockerfile

# Start with an official PyTorch image as a base
FROM pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime

# Install any additional libraries and tools needed
RUN apt-get update && apt-get install -y --no-install-recommends \
libgomp1
RUN apt-get install -y libpq-dev gcc

# &&
# rm -rf /var/lib/apt/lists/*

# Install the necessary Python packages
COPY requirements_sm_training.txt /opt/ml/code/requirements.txt
RUN pip install --no-cache-dir -r /opt/ml/code/requirements.txt

# Probably don't need this
RUN pip3 install sagemaker-training

# Copy the training script
COPY train/train_illuvitar_neural_net.py /opt/ml/code/train
RUN chmod +x /opt/ml/code/train
COPY shared/ /opt/ml/code/shared/
COPY helper/ /opt/ml/code/helper/

# Set the working directory
WORKDIR /opt/ml/code
ENV PATH="/opt/ml/code:${PATH}"


Training script arg parser

def get_args():
parser = argparse.ArgumentParser()

is_local = os.environ.get("ISLOCAL")
default_model_dir = (
"./local_model_storage" if is_local else "/opt/ml/model"
) # os.environ.get("SM_MODEL_DIR")

parser.add_argument("--epochs", type=int, default=500, help="Number of training epochs")
parser.add_argument("--learning_rate", type=float, help="Learning rate for the optimizer")
parser.add_argument("--batch_size", type=int, default=1024, help="Batch size for training")
parser.add_argument("--model_dir", type=str, default=default_model_dir)
parser.add_argument("--input_size", type=int, default=57, help="Input size for the model")
parser.add_argument(
"--hidden_size", type=int, default=40, help="Number of neurons in the hidden layer"
)
parser.add_argument("--output_size", type=int, default=1, help="Output size for the model")
parser.add_argument("--patience", type=int, default=200, help="Patience for early stopping")
parser.add_argument("--min_delta_for_early_stopping", type=float, default=0.03)
parser.add_argument("--hidden_layer_count", type=int, default=3, help="Number of hidden layers")
parser.add_argument(
"--is_alpha", type=bool, default=False, help="Is this an alpha model?"
) # sagemaker doesn't support actions lik "store_true"
parser.add_argument("--training_job_name", type=str, default="sagemaker-run-1684176842-32dd")
parser.add_argument("--experiment_name", type=str, default="firstexperiment")

parser.add_argument("--run_number", type=int, default=1)

args = parser.parse_args()

return args


What do I need to modify to get this to work? Or am I fundamentally misunderstanding something here?

I can't for the life of me find an example of this anywhere, anytime an example has a custom image it doesn't have hyperparameters specified, any example with hyperparameters doesn't use a custom image.

Continue reading...