[Python] AWS SageMaker Auto Scaling Not Triggering Despite CloudWatch Alarm Being in "In...

I have set up automatic scaling for my SageMaker endpoint variant using CloudWatch alarms and Application Auto Scaling, but the scaling action is not being triggered despite the alarm being in the "In alarm" state.

SageMaker Endpoint:

Endpoint name: asynchronous-inference-test-autoscaling-dev-7 Type: Asynchronous

CloudWatch Metric:

Metric Name: HasBacklogWithoutCapacity Condition: The alarm triggers if the metric value is ≥1 for 1 datapoint within 10 seconds.

Scaling Policy:

Scale Up Policy: The policy scales up the instance count by 1 when the HasBacklogWithoutCapacity metric is ≥1. Scale Down Policy: The policy scales down the instance count to 0 when the HasBacklogWithoutCapacity metric is 0.

Alarm Actions:

The alarm ScaleUpTo1InstanceAlarm is in the "In alarm" state, indicating that the metric HasBacklogWithoutCapacity has exceeded the threshold (≥1), but the scaling action is not being triggered.

Configuration Details:

The cooldown period is set to 30 seconds. The alarm is set to evaluate the metric every 10 seconds and triggers the scaling policy when HasBacklogWithoutCapacity is ≥1.

Thank you for your time.

CODE BELOW

import boto3

# resource_id for my SageMaker endpoint variant
resource_id = f"endpoint/{endpoint_name}/variant/{variant_name}"

client = boto3.client('application-autoscaling')

# Register the scalable target for the endpoint variant
client.register_scalable_target(
ServiceNamespace='sagemaker',
ResourceId=resource_id,
ScalableDimension='sagemaker:variantesiredInstanceCount',
MinCapacity=0,
MaxCapacity=1
)

# Create the scaling policy to scale up when HasBacklogWithoutCapacity >= 1
response_scale_up = client.put_scaling_policy(
PolicyName="ScaleUpTo1InstancePolicy",
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variantesiredInstanceCount",
PolicyType="StepScaling",
StepScalingPolicyConfiguration={
"AdjustmentType": "ChangeInCapacity",
"MetricAggregationType": "Average",
"Cooldown": 30, # Cooldown period (in seconds)
"StepAdjustments": [
{
"MetricIntervalLowerBound": 0,
"ScalingAdjustment": 1 # Scale up by 1 instance
}
]
},
)

# Extract the Policy ARN
policy_arn_scale_up = response_scale_up["PolicyARN"]
print("Scale-up policy ARN:", policy_arn_scale_up)

CloudWatch client for alarm setup


cw_client = boto3.client("cloudwatch")

response_alarm_scale_up = cw_client.put_metric_alarm(
AlarmName="ScaleUpTo1InstanceAlarm",
MetricName="HasBacklogWithoutCapacity", # The CloudWatch metric to monitor
Namespace="AWS/SageMaker",
Statistic="Average",
EvaluationPeriods=1, # Evaluate the metric once
DatapointsToAlarm=1, # Trigger alarm if condition is met in 1 datapoint
Threshold=1, # Threshold to trigger scale-up
ComparisonOperator="GreaterThanOrEqualToThreshold",
TreatMissingData="breaching", # Treat missing data as breaching threshold
Dimensions=[
{'Name': 'EndpointName', 'Value': endpoint_name}
],
Period=10, # Period to evaluate the metric (in seconds)
AlarmActions=[policy_arn_scale_up] # Link the alarm to the scaling policy
)

print("Scale-up alarm created:", response_alarm_scale_up)

# Create the scaling policy to scale down to 0 instances when HasBacklogWithoutCapacity = 0
response_scale_down = client.put_scaling_policy(
PolicyName="ScaleDownToZeroInstancePolicy",
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variantesiredInstanceCount",
PolicyType="StepScaling",
StepScalingPolicyConfiguration={
"AdjustmentType": "ChangeInCapacity",
"MetricAggregationType": "Average", # Use average of the metric
"Cooldown": 30, # Cooldown period (in seconds)
"StepAdjustments": [
{
"MetricIntervalLowerBound": 0, # Scale down when metric is 0
"ScalingAdjustment": -1 # Scale down by 1 instance
}
]
},
)

# Extract the Policy ARN
policy_arn_scale_down = response_scale_down["PolicyARN"]
print("Scale-down policy ARN:", policy_arn_scale_down)

# CloudWatch alarm for scale down
response_alarm_scale_down = cw_client.put_metric_alarm(
AlarmName="ScaleDownToZeroInstanceAlarm",
MetricName="HasBacklogWithoutCapacity", # The CloudWatch metric to monitor
Namespace="AWS/SageMaker",
Statistic="Average",
EvaluationPeriods=1, # Evaluate the metric once
DatapointsToAlarm=1, # Trigger alarm if condition is met in 1 datapoint
Threshold=0, # Threshold to trigger scale-down
ComparisonOperator="LessThanOrEqualToThreshold",
TreatMissingData="ignore", # Ignore missing data (won't trigger scale-up)
Dimensions=[
{'Name': 'EndpointName', 'Value': endpoint_name}
],
Period=10, # Period to evaluate the metric (in seconds)
AlarmActions=[policy_arn_scale_down] # Link the alarm to the scaling policy
)

print("Scale-down alarm created:", response_alarm_scale_down)

Continue reading...