Automating EC2 Start/Stop Schedules to Cut Non-Prod Costs

#Operational requirement

Non-production AWS environments — dev, test, staging, sandbox — run around the clock the moment someone launches an EC2 instance without an attached stop schedule. Nobody switches these off by hand consistently. They forget, they go on leave, they change teams. On one graduate placement I ran, a single unattended sandbox account accumulated close to £14,000 in EC2 and EBS charges over a six-week sprint because three m5.2xlarge instances sat idle overnight and across weekends for two months straight. Finance caught it before engineering did. That is the failure you are being asked to close down.

Your assignment, usually handed to a new engineer in the first fortnight of a placement, is to build an automated start/stop schedule for non-production EC2 instances using resource tags, Amazon EventBridge scheduled rules and a Lambda function. Two conditions apply without exception: production must never be touched by this automation, and nothing that genuinely needs to run outside office hours gets shut down by accident.

Treat what you build as production infrastructure, even though it only manages non-production resources. It has to survive account handovers, instance replacements and staff turnover without anyone remembering to maintain it by hand. If it needs a person to remember something every month, it will eventually fail silently.

#Prerequisites and permissions

Confirm the following before you touch the console. Missing any one of these is the usual reason a first attempt stalls halfway through a sprint.

• AWS CLI v2 configured with a named profile scoped to the non-production account, never the management account.
• IAM permissions to create a Lambda function, an execution role and EventBridge rules, granted through a scoped permission set rather than AdministratorAccess.
• Write access to tag existing EC2 instances, or a change ticket authorising a platform engineer to do it on your behalf.
• Sign-off from the environment owner on the tagging schema and the exact start and stop times. Do not invent business hours yourself.
• A test instance you are explicitly permitted to stop and start repeatedly during development, isolated from anything a QA team is actively using.

Confirm your CLI identity before running anything containing a stop or start action:

1aws sts get-caller-identity --profile nonprod

If that command returns an account number you do not recognise, stop immediately. Do not proceed until you have confirmed you are pointed at the correct account. This one check has prevented more than one accidental production shutdown.

#Staged implementation

#Stage 1: lock down the tagging convention before writing any code

Every automated shutdown control lives or dies on tag discipline. Agree the schema with the platform lead first, then enforce it once it is stable. A workable schema:

1Key: Environment   Value: dev | test | staging
2Key: Schedule      Value: weekday-0700-1900 | weekday-0800-2000 | none
3Key: Owner         Value: team-alias

Tag existing instances before deploying any automation:

1aws ec2 create-tags 
2  --resources i-0123456789abcdef0 
3  --tags Key=Environment,Value=dev Key=Schedule,Value=weekday-0700-1900

Never set Schedule=none unless there is a documented business justification signed off by the environment owner. That value is how instances quietly opt out of cost controls forever, and six months later nobody remembers why. Once the schema is stable, enforce it with an AWS Config rule so drift gets flagged automatically:

1aws configservice put-config-rule 
2  --config-rule '{
3    "ConfigRuleName": "nonprod-required-tags",
4    "Source": {"Owner": "AWS", "SourceIdentifier": "REQUIRED_TAGS"},
5    "InputParameters": "{"tag1Key":"Environment","tag2Key":"Schedule"}"
6  }'

#Stage 2: create a least-privilege IAM execution role

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Action": ["ec2:DescribeInstances", "ec2:StartInstances", "ec2:StopInstances"],
7      "Resource": "*",
8      "Condition": {
9        "StringEquals": { "aws:ResourceTag/Environment": ["dev", "test", "staging"] }
10      }
11    },
12    {
13      "Effect": "Allow",
14      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
15      "Resource": "arn:aws:logs:*:*:*"
16    }
17  ]
18}

The Condition block is the control that stops this Lambda from ever touching a production-tagged instance, even if someone fat-fingers an EventBridge target later. Do not skip it because the Lambda only targets one account today. Accounts get merged, roles get reused, and that assumption ages badly.

#Stage 3: deploy the Lambda function

1import boto3
2
3ec2 = boto3.client("ec2")
4
5def handler(event, context):
6    action = event.get("action")
7    filters = [
8        {"Name": "tag:Schedule", "Values": ["weekday-0700-1900", "weekday-0800-2000"]},
9        {"Name": "instance-state-name", "Values": ["running" if action == "stop" else "stopped"]}
10    ]
11    reservations = ec2.describe_instances(Filters=filters)["Reservations"]
12    ids = [i["InstanceId"] for r in reservations for i in r["Instances"]]
13    if not ids:
14        return {"processed": 0}
15    if action == "stop":
16        ec2.stop_instances(InstanceIds=ids)
17    else:
18        ec2.start_instances(InstanceIds=ids)
19    return {"processed": len(ids)}

Pass the action through the EventBridge input transformer rather than writing two nearly identical functions. One function and two rules means less drift to manage during on-call handovers.

#Stage 4: wire up EventBridge scheduled rules

1aws events put-rule 
2  --name nonprod-ec2-stop-1900-utc 
3  --schedule-expression "cron(0 19 ? * MON-FRI *)"
4
5aws events put-rule 
6  --name nonprod-ec2-start-0700-utc 
7  --schedule-expression "cron(0 7 ? * MON-FRI *)"
8
9aws lambda add-permission 
10  --function-name nonprod-ec2-scheduler 
11  --statement-id eventbridge-invoke-stop 
12  --action lambda:InvokeFunction 
13  --principal events.amazonaws.com 
14  --source-arn arn:aws:events:eu-west-2:111122223333:rule/nonprod-ec2-stop-1900-utc
15
16aws events put-targets 
17  --rule nonprod-ec2-stop-1900-utc 
18  --targets "Id"="1","Arn"="arn:aws:lambda:eu-west-2:111122223333:function:nonprod-ec2-scheduler","Input"="{"action":"stop"}"

Use cron expressions in UTC deliberately. Do not hardcode a local timezone offset. British Summer Time will shift a "7am start" to 8am for half the year, and nobody notices until the QA team logs a complaint about environments being unavailable when they arrive.

#Stage 5: run a pilot before enrolling the whole estate

Do not point the schedule at every tagged instance on day one. Restrict the Lambda filter to two or three low-risk instances for the first week, watch the logs and Cost Explorer figures daily, and only widen the tag filter once you have a full week of clean stop and start cycles with no exceptions raised by any team.

#Choosing an approach

For a graduate-scale account or a single business unit, build the custom Lambda path. It is auditable, small enough for one engineer to fully understand, and cheap to run. Escalate to the published Instance Scheduler solution once you are managing more than roughly fifteen accounts, because per-account rule sprawl becomes unmanageable by hand.

#Verification

Confirm the rule fired and did what you expect before you consider the change complete. Check the Lambda logs first:

1aws logs filter-log-events 
2  --log-group-name /aws/lambda/nonprod-ec2-scheduler 
3  --start-time $(date -d "-1 hour" +%s000)
4
5aws ec2 describe-instances 
6  --filters "Name=tag:Schedule,Values=weekday-0700-1900" 
7  --query "Reservations[].Instances[].[InstanceId,State.Name]" 
8  --output table

Check the rule’s own delivery metrics too. A rule can be enabled and correctly targeted but silently fail every invocation if the Lambda permission was never granted:

1aws cloudwatch get-metric-statistics 
2  --namespace AWS/Events 
3  --metric-name FailedInvocations 
4  --dimensions Name=RuleName,Value=nonprod-ec2-stop-1900-utc 
5  --start-time 2025-01-01T00:00:00Z --end-time 2025-01-02T00:00:00Z 
6  --period 3600 --statistics Sum

After the first full week, pull a tag-filtered Cost Explorer report against the Environment tag and confirm the run-hours dropped roughly in line with the schedule. If the number barely moves, something is still exempting instances, usually an Auto Scaling Group relaunching them, covered next.

#Failure Modes

The Auto Scaling Group trap catches almost every junior engineer once. If an instance sits inside an ASG, stopping it directly fights the ASG’s own reconciliation loop. It will either relaunch a replacement or mark the stopped instance unhealthy and terminate it. Non-production ASGs should either be excluded from this automation entirely or scaled to zero on a schedule instead of having individual instances stopped.

#Monitoring and change control

Set a CloudWatch alarm on the FailedInvocations metric for both rules and route it to the team’s SNS topic. Silent failures are the real risk here — nobody complains when an instance stays running, but everyone complains when one goes down unexpectedly.

1aws cloudwatch put-metric-alarm 
2  --alarm-name nonprod-scheduler-failed-invocations 
3  --namespace AWS/Events 
4  --metric-name FailedInvocations 
5  --dimensions Name=RuleName,Value=nonprod-ec2-stop-1900-utc 
6  --statistic Sum --period 3600 --threshold 1 
7  --comparison-operator GreaterThanOrEqualToThreshold 
8  --evaluation-periods 1 
9  --alarm-actions arn:aws:sns:eu-west-2:111122223333:platform-alerts

Log every schedule addition, tag exemption or window change through the standard change ticket, referencing the instance IDs and the reason. Run a quarterly tag audit against the Config rule report to catch instances that drifted out of scope after replacement or migration. Any instance found untagged during that audit should be treated as an incident, not a minor cleanup item, because it means the cost control has a gap nobody noticed.

#Rollback

If the schedule causes an incident — a demo environment goes dark mid-presentation, or a dependent batch job fails because its host stopped mid-run — the safe reversal path is immediate and does not require deleting anything:

1aws events disable-rule --name nonprod-ec2-stop-1900-utc
2aws events disable-rule --name nonprod-ec2-start-0700-utc
3
4aws ec2 start-instances --instance-ids i-0123456789abcdef0

Disabling the rules leaves the Lambda function, IAM role and tag schema intact, so you can re-enable once the root cause is fixed rather than rebuilding from scratch. Do not delete the EventBridge rules as a first response. Disabling is reversible in seconds; deleting means re-authoring cron expressions and target bindings under pressure, which is exactly when mistakes happen.

Once the incident is resolved, re-enable both rules in the same order you disabled them, verify with the CloudWatch Logs query from the Verification section, and log the change in the change record with the instance IDs affected and the duration of the exemption.

#Operational Summary

This control is small enough to build in an afternoon and consequential enough to matter every day it runs. The tagging convention is the actual control; the Lambda function is just the enforcement mechanism, so get the schema agreed and documented before writing any code. Pilot on a handful of instances first, verify with logs and Cost Explorer rather than a single glance at the console, and set an alarm on failed invocations so silence does not mean success by default. Watch for Auto Scaling Groups and RDS dependencies specifically — they are the two most common reasons this automation appears to work in testing and then causes an incident in week three. Keep the rollback path to a disable, not a delete, and log every exemption through change control. Handled this way, the next graduate engineer inherits a control they can trust rather than a set of rules nobody dares to touch.