Simulation Notice
This is a fictional, bounded simulation created for diagnostic-reasoning practice. No real incident, live telemetry, credentials or production identifiers are involved. All names, account references and metrics are invented for this exercise.
Scenario
Northgate Retail, a fictional online retailer, runs its checkout service on Amazon Web Services behind an Application Load Balancer (ALB) fronting an Auto Scaling group of EC2 instances in a single Region. At 14:20 on a fictional Tuesday, on-call engineer "Priya", a fictional persona, receives an alert: the ALB error rate has risen sharply, and customer support is fielding reports of intermittent checkout failures. CPU utilisation across the Auto Scaling group looks unremarkable. Priya must decide whether this is an application fault, an infrastructure capacity issue, or a downstream dependency problem before taking any action.
Evidence
- The ALB target group shows three of six targets marked "unhealthy" over a ten-minute window, cycling in and out of service.
- The ALB's target 5xx error count rose from a fictional baseline of roughly two per minute to roughly forty per minute.
- The Auto Scaling group's average CPU utilisation remains at a fictional 35 percent, well below any scale-out threshold.
- The target group health check path returns HTTP 200 when tested manually from a bastion host inside the same VPC.
- Application logs on the affected instances show repeated "connection timeout" entries referencing a fictional internal payment-authorisation endpoint, not the health check path itself.
- No recent deployment, AMI change or Auto Scaling configuration change appears in the fictional change log for the preceding twenty-four hours.
Hypotheses and Actions
Priya considers three candidate hypotheses before acting.
- Hypothesis A, capacity exhaustion: instances are overloaded and failing health checks under load. This is weakened by the flat CPU utilisation evidence.
- Hypothesis B, health check misconfiguration: the ALB health check itself is faulty. This is weakened because manual checks from inside the VPC return 200 successfully.
- Hypothesis C, downstream dependency failure: a dependency, the fictional payment-authorisation endpoint, is timing out, exhausting request threads and causing the instances to fail health checks indirectly. This is supported by the application log evidence.
Rather than immediately restarting instances or forcing a scale-out event, which would treat the symptom of unhealthy targets without addressing a suspected cause, Priya gathers further read-only evidence to test Hypothesis C before recommending any change.
Progressive Hints
- Hint 1: compare the metric that triggered the alert against a metric that would move if capacity were the true cause.
- Hint 2: check whether the health check endpoint and the failing application code path are the same route.
- Hint 3: look for anything that changed outside the service's own boundary, not only within it.
Reveal
In this fictional scenario, the payment-authorisation endpoint's security group was inadvertently tightened by an unrelated fictional pipeline change, restricting inbound access from the checkout service. Requests from checkout instances timed out waiting for a response, the request-handling thread pool filled, and the ALB's health check requests, queued behind the same worker pool, then failed intermittently. This produced symptoms that closely resembled an infrastructure capacity problem. The correct fix was scoped to restoring the appropriate security group rule for the dependency, not to the checkout service's Auto Scaling group or ALB health check configuration.
Learning Outcome
This exercise demonstrates that a flat capacity metric can rule out one hypothesis while a shared resource, such as a thread pool, can make an unrelated dependency failure masquerade as an infrastructure symptom. Verifying evidence against each hypothesis before acting, rather than reacting to the most visible symptom, is the transferable diagnostic skill. The next safe decision in a real environment would be to confirm the dependency's security group rules against a documented baseline, restore least-privilege access deliberately, and monitor target health for a stable recovery window before closing the investigation.