Enforcing AWS Auto Scaling Health Checks for Resilient EC2

#Operational requirement

A newly hired engineer will inherit Auto Scaling groups that were built once, during a migration sprint, and never touched again. The default configuration relies on EC2 status checks only, which means an Auto Scaling group will happily keep an instance in service even when the application inside it has stopped responding on port 443, the web process has crashed, or a dependency like a local cache has wedged. EC2 status checks only detect hypervisor-level failure: a stopped instance, a failed network interface, a hardware fault. They do not detect an application that is technically running but returning HTTP 502s to every customer. If your Auto Scaling group is fronted by an Application Load Balancer and you have not explicitly enabled ELB health checks with a sensible grace period, you are running production on a false sense of security.

The production consequence is straightforward and expensive: a bad deployment or a memory leak takes down one or two instances behind the load balancer, the ASG considers them perfectly healthy because the EC2 status check passes, and traffic keeps flowing to dead instances until a human notices error rates climbing in CloudWatch or, worse, until a customer complains. This tutorial walks through converting an existing Auto Scaling group from EC2-only health checks to combined EC2 and ELB health checks, tuning the health check grace period correctly, and proving the failover actually works before you hand it back to the team.

#Prerequisites and required permissions

You will need an IAM identity (user or role) with at minimum autoscaling:DescribeAutoScalingGroups, autoscaling:UpdateAutoScalingGroup, elasticloadbalancing:DescribeTargetHealth, elasticloadbalancing:ModifyTargetGroup, and cloudwatch:GetMetricData permissions. In most enterprise AWS accounts this will be scoped through a permission set assigned via AWS IAM Identity Center rather than a static IAM user; do not create a long-lived access key for this task, and do not proceed if your session token has fewer than thirty minutes of validity remaining, because a mid-change expiry leaves the group in a partially updated state.

• An existing Auto Scaling group attached to an Application Load Balancer target group, with at least two running instances in service
• AWS CLI v2 configured with a named profile pointing at the target account and region
• Change management ticket approved: this is a production-affecting configuration change, not a read-only audit
• A maintenance window or agreed low-traffic period, since you will deliberately fail a health check to prove behaviour
• Session Manager access to at least one instance in the group, or SSH key access if Session Manager is not yet deployed
• Familiarity with your application’s real startup time, measured, not guessed

Confirm your CLI access and current configuration before touching anything:

1aws sts get-caller-identity --profile prod-eng
2aws autoscaling describe-auto-scaling-groups 
3  --auto-scaling-group-names web-tier-asg 
4  --profile prod-eng 
5  --query "AutoScalingGroups[0].[HealthCheckType,HealthCheckGracePeriod]"

If that query returns ["EC2", 300], you have found exactly the gap this tutorial closes.

#Step 1: Baseline the current configuration and capture evidence

Action: pull the full current state of the Auto Scaling group and its associated target group health before changing anything. This is your rollback reference and your change ticket attachment, and it is not optional.

1aws autoscaling describe-auto-scaling-groups 
2  --auto-scaling-group-names web-tier-asg 
3  --profile prod-eng > baseline-asg-config.json
4
5aws elbv2 describe-target-health 
6  --target-group-arn arn:aws:elasticloadbalancing:eu-west-2:111122223333:targetgroup/web-tier-tg/abc123 
7  --profile prod-eng > baseline-target-health.json

Expected result: two JSON files showing HealthCheckType: EC2, the current HealthCheckGracePeriod, and a target health list where every registered target shows State: healthy. Evidence to capture: attach both JSON files to the change ticket with a timestamp. Checkpoint: do not proceed until you can explain, in one sentence, why the current setup would fail to detect an application-level outage. If you cannot articulate that, stop and re-read the operational requirement.

#Step 2: Enable ELB health checks with a correct grace period

Action: update the Auto Scaling group’s HealthCheckType to ELB and set HealthCheckGracePeriod to a value that comfortably exceeds your application’s cold-start time, typically sixty to ninety seconds past your average boot-to-ready time, not the AWS default of 300 seconds unless your application genuinely needs it.

1aws autoscaling update-auto-scaling-group 
2  --auto-scaling-group-name web-tier-asg 
3  --health-check-type ELB 
4  --health-check-grace-period 180 
5  --profile prod-eng

Expected result: the command returns no output on success, which is the AWS CLI convention for this API call. Verify immediately with a describe call rather than trusting silence. Evidence to capture: the before and after diff of HealthCheckType and HealthCheckGracePeriod. Checkpoint: confirm the target group’s own health check path is meaningful; checking / is not sufficient if your application has a dedicated /health endpoint that verifies downstream dependencies.

1aws elbv2 modify-target-group 
2  --target-group-arn arn:aws:elasticloadbalancing:eu-west-2:111122223333:targetgroup/web-tier-tg/abc123 
3  --health-check-path /health 
4  --health-check-interval-seconds 15 
5  --healthy-threshold-count 2 
6  --unhealthy-threshold-count 3 
7  --profile prod-eng

#Step 3: Prove failover by deliberately breaking one instance

Action: during the agreed maintenance window, connect via Session Manager and stop the application process, not the instance, to simulate an application-level failure while the underlying EC2 status remains healthy throughout.

1aws ssm start-session --target i-0abc123def456789 --profile prod-eng
2# inside the session
3sudo systemctl stop nginx

Expected result: within roughly unhealthy-threshold-count multiplied by health-check-interval-seconds, around 45 seconds in this configuration, the target group marks that instance unhealthy, and shortly afterwards the Auto Scaling group terminates it and launches a replacement. Evidence to capture: CLI output of describe-target-health showing the transition from healthy to unhealthy, and the Auto Scaling activity history showing a terminate-and-launch cycle.

1aws autoscaling describe-scaling-activities 
2  --auto-scaling-group-name web-tier-asg 
3  --max-items 5 
4  --profile prod-eng

Checkpoint: if the instance never flips to unhealthy, check the security group on the target first. A missing inbound rule for the load balancer’s health check traffic is the most common cause, and it will not throw an error; it will simply time out silently, which is far worse for diagnosis.

#Verification

Retain the following as your completion evidence pack: the baseline JSON files from Step 1, the updated describe-auto-scaling-groups output showing HealthCheckType: ELB, the target health transition log from Step 3, and the CloudWatch metric graph for GroupInServiceInstances across the test window. This is the flow you should be able to explain to your lead without notes:

A verification pass is incomplete without a second data point: check the ALB’s own TargetResponseTime and HTTPCode_Target_5XX_Count metrics for the same window, and confirm the 5XX count drops to zero once the replacement instance is registered. A healthy target group with a non-zero 5XX count is a sign that the health check path is too shallow, not that the ASG configuration is wrong.

#Failure Modes or Common Traps

The single most common trap is setting HealthCheckGracePeriod too short relative to actual application startup time. If your application takes 120 seconds to warm a cache before serving traffic and your grace period is 60 seconds, the ASG will terminate perfectly good instances in an endless churn loop, and you will see GroupInServiceInstances oscillating in CloudWatch with no obvious root cause and a growing instance launch bill.

A second trap: forgetting that changing HealthCheckType to ELB without also verifying the target group’s health check configuration means you inherit whatever thresholds were set, sometimes deliberately loose, by whoever built the load balancer originally. Always check both resources together; never assume they were configured as a pair.

Security groups are the silent killer here. If the target group cannot reach the instance on the health check port, the target sits in unhealthy state indefinitely with nothing more descriptive than a generic request-timed-out message in the console. There is no distinct AWS error code beyond that generic timeout, so check network reachability first, not last, or you will waste an afternoon staring at CloudWatch graphs that show nothing.

Do not run this test against a group with only one instance in service. You will cause a genuine outage rather than a controlled failover demonstration, and the change ticket will not save you from that conversation with the service owner.

A subtler failure than outright churn is a target group that reports healthy in describe-target-health while the application still returns intermittent errors to real users. This typically means the health check path returns a shallow response without exercising the same code path or dependency as genuine traffic. Distinguish this from a genuine misconfiguration by comparing the target group’s ALB response-code metrics against actual customer-reported errors; if the discrepancy persists, the health check endpoint itself needs remediation as separate follow-up work, outside the scope of this configuration change.

#Rollback

If the change causes unexpected termination churn, or the on-call engineer needs to revert immediately during an incident, restore the exact baseline captured in Step 1:

1aws autoscaling update-auto-scaling-group 
2  --auto-scaling-group-name web-tier-asg 
3  --health-check-type EC2 
4  --health-check-grace-period 300 
5  --profile prod-eng

Then revert the target group health check path and thresholds if they were changed:

1aws elbv2 modify-target-group 
2  --target-group-arn arn:aws:elasticloadbalancing:eu-west-2:111122223333:targetgroup/web-tier-tg/abc123 
3  --health-check-path / 
4  --health-check-interval-seconds 30 
5  --healthy-threshold-count 5 
6  --unhealthy-threshold-count 2 
7  --profile prod-eng

Confirm rollback with the same describe commands used in Step 1 and attach the output to the change ticket as evidence of reversal. Do not close the ticket until the target group shows every instance as healthy under the reverted configuration; a rollback that leaves an instance stuck mid-transition is not a completed rollback.

#Operational Summary

#Monitoring and alerting configuration

Health check conversion without corresponding alarm coverage leaves the on-call rotation blind to churn. Create a CloudWatch alarm against GroupInServiceInstances and a second against the target group’s UnHealthyHostCount metric, both scoped to the specific target group dimension rather than the load balancer as a whole.

1aws cloudwatch put-metric-alarm 
2  --alarm-name web-tier-unhealthy-host-count 
3  --namespace AWS/ApplicationELB 
4  --metric-name UnHealthyHostCount 
5  --dimensions Name=TargetGroup,Value=targetgroup/web-tier-tg/abc123 Name=LoadBalancer,Value=app/web-tier-alb/xyz789 
6  --statistic Average 
7  --period 60 
8  --threshold 1 
9  --comparison-operator GreaterThanOrEqualToThreshold 
10  --evaluation-periods 3 
11  --alarm-actions arn:aws:sns:eu-west-2:111122223333:oncall-paging 
12  --profile prod-eng

Set the evaluation window to at least three periods to avoid paging on a single transient blip that the unhealthy threshold count would have absorbed anyway. Confirm the alarm reaches OK state within a few minutes of creation; an alarm stuck in INSUFFICIENT_DATA usually means the dimension values do not exactly match the ARN suffix format CloudWatch expects, which is a common copy-paste error when scripting this against multiple environments.

#

Escalation thresholds

Define a clear escalation boundary in the runbook handover. A single instance cycling through unhealthy-terminate-replace once is expected behaviour and requires no page. Three or more termination events against the same Auto Scaling group within a fifteen-minute window indicates either a bad deployment or a misconfigured grace period, and should escalate to the deploying engineer immediately rather than waiting for a customer-facing symptom. Sustained oscillation beyond thirty minutes with no stabilisation should escalate to the service owner and trigger consideration of a full rollback, not further tuning under pressure.

#Change-control record keeping

Every parameter change described in this runbook is a production change and must be logged against the change ticket with explicit before and after values, not a narrative description written after the fact. At minimum, record the ticket number, the engineer’s IAM principal from get-caller-identity, the exact CLI commands executed with timestamps, and the maintenance window agreed with the service owner. Where the change management system supports it, attach the baseline JSON files as immutable evidence rather than pasting truncated excerpts into a comment field, since auditors reviewing an incident months later will want the raw describe output, not a summary written from memory.

You have converted an Auto Scaling group from a hypervisor-only health signal to one that reflects genuine application availability, tuned the grace period against real startup behaviour, and proven the failover mechanism under controlled conditions rather than trusting it blindly. Retain the evidence pack, baseline JSON, transition logs, activity history, because the next production incident review will ask exactly what health check behaviour was in place at the time. For deeper reference, consult the AWS documentation on Amazon EC2 Auto Scaling health checks and the AWS documentation on Application Load Balancer target group health checks. As a final assessment, be ready to explain to your lead, without notes, why an instance can pass an EC2 status check while failing every real customer request; that gap is precisely what this configuration closes.