Automated EC2 Patch Compliance with AWS Systems Manager
A hands-on runbook for junior engineers configuring AWS Systems Manager Patch Manager with staged rollout, compliance evidence and snapshot rollback.
Priya NairIn this lesson
Table of Contents
Table of contents
Before you begin
- Basic AWS CLI usage and IAM role concepts
- Familiarity with EC2 instances and EBS volumes
- Understanding of change management/CAB approval processes
Track this tutorial
Choose your current status and tick each safety check as you complete it. Sign in to sync progress between devices.
Current status
Tutorial stages
- Step 1: Confirm the fleet is SSM-managed
- Step 2: Select or create the Patch Baseline
- Step 3: Tag instances and register the Patch Group
- Step 4: Establish a pre-patch snapshot policy
- Step 5: Create the Maintenance Window and register the scan-only task
- Step 6: Promote to Install mode with change approval
0 of 6 stages complete
Before you apply the change
Confirm these production-safety controls during the tutorial.
#Operational requirement
You have just been handed the app-tier fleet with an instruction that sounds simple and is not: “get patching under control and show me proof by Friday.” Unpatched EC2 instances are the most common root cause you will find written into a post-incident review with your name in the timeline. A missed critical CVE on a public-facing box is not an abstract risk on a slide — it is the difference between a scheduled maintenance window and an out-of-hours page from the security team telling you an instance is talking to an address in a country your business has no reason to touch.
Manually SSHing into boxes to run yum update or apt upgrade does not scale past a handful of hosts, leaves nothing for a change auditor to inspect, and gives you no evidence when a client security questionnaire asks how patch currency is proven across the estate. This is the standard operating procedure for standing up AWS Systems Manager Patch Manager against a production fleet: scan first, gate behind change approval, install second, and keep a rollback path you have actually rehearsed rather than assumed will work. Follow this every time you are handed a new fleet to bring under compliance — do not treat it as a one-off project.
#Prerequisites and required permissions
Confirm every item below before you touch the production account. If anything is missing, stop and escalate to your lead rather than improvising permissions on the fly.
- IAM rights to create Patch Baselines, Maintenance Windows and pass roles:
ssm:CreatePatchBaseline,ssm:CreateMaintenanceWindow,ssm:RegisterTargetWithMaintenanceWindow,ssm:RegisterTaskWithMaintenanceWindow,iam:PassRole. - Every target instance has the SSM Agent running and an attached instance profile including
AmazonSSMManagedInstanceCore. An instance without a matching role never appears as “Managed” in Fleet Manager, and Patch Manager silently skips it — no error, just absence. - An outbound path to the SSM endpoints, either through a NAT gateway or, for private subnets with no NAT, VPC interface endpoints for
ssm,ssmmessagesandec2messages. This is a very common gap in locked-down production VPCs and the first thing to check when instances go dark. - A tagging standard already agreed with the platform team, specifically a
Patch Grouptag on every instance, for examplePatch Group = production-webtier. Patch Manager routes baselines by this tag, never by instance ID. - A change record raised and approved through your CAB or equivalent tooling before the first Install-mode run against production. Scan-only runs are non-disruptive and typically do not require CAB sign-off under most change frameworks, but confirm this against your own organisation’s policy rather than assuming it.
- AWS CLI v2 configured with a named profile scoped to the target account, or CloudShell access if CLI installation is restricted on your workstation.
- An owner named for the CloudWatch alarm that will watch compliance drift once this is live — do not build the pipeline and leave nobody accountable for the alerts it generates.
#Step 1: Confirm the fleet is SSM-managed
Run this against the target account and region first, before you configure anything:
1aws ssm describe-instance-information
2 --query "InstanceInformationList[*].[InstanceId,PingStatus,PlatformType,AgentVersion]"
3 --output tableExpected result: every intended target shows PingStatus: Online. Any production instance missing from this list cannot be patched regardless of tagging — fix the IAM role or the agent service first, and do not proceed until the gap is closed.
Capture this table into the change ticket as your pre-implementation baseline of managed instance count. That number becomes the figure you reconcile against at every later checkpoint.
#Step 2: Select or create the Patch Baseline
Default to the AWS-provided baseline for the OS family unless your organisation runs a documented exception policy, for example deferring non-critical patches by seven days to avoid vendor day-zero regressions. To create a custom baseline with that delay for Amazon Linux
1aws ssm create-patch-baseline
2 --name "prod-al2-baseline"
3 --operating-system AMAZON_LINUX_2
4 --approval-rules '{"PatchRules":[{"PatchFilterGroup":{"PatchFilters":[{"Key":"CLASSIFICATION","Values":["Security","Bugfix"]}]},"ApproveAfterDays":7,"ComplianceLevel":"CRITICAL"}]}'Record the returned BaselineId and reference it explicitly in the maintenance window task, rather than relying on the account default. An explicit reference is safer — it will not silently change if another team edits the account-wide default later. Confirm the classification filters match policy before moving on: a baseline scoped only to Security will quietly skip bugfix patches your compliance dashboard still reports on, and you will not notice until an auditor does.
#Step 3: Tag instances and register the Patch Group
1aws ec2 create-tags --resources i-0123456789abcdef0
2 --tags Key=Patch Group,Value=production-webtier
3
4aws ssm register-patch-baseline-for-patch-group
5 --baseline-id pb-0abc123def456ghi7
6 --patch-group production-webtierConfirm the association with aws ssm describe-patch-group-state --patch-group production-webtier, and check the returned instance count against Step 1’s figure. A mismatch almost always means a tagging typo — the key is case-sensitive and space-sensitive, and “Patch Group” with a trailing space is a different key entirely from AWS’s perspective.
#Step 4: Establish a pre-patch snapshot policy
Before any Install-mode run, a recent EBS snapshot must exist for every target volume. Do not skip this because it is “just a patch run” — kernel and glibc updates have broken boot on production fleets before, and a two-minute volume restore is a very different afternoon from a rebuild-from-AMI.
1aws ec2 create-snapshot
2 --volume-id vol-0a1b2c3d4e5f67890
3 --description "pre-patch-2024-06-prod-webtier"
4 --tag-specifications 'ResourceType=snapshot,Tags=[{Key=PatchRollback,Value=true}]'Better practice is to automate this as a pre-task ahead of the maintenance window using an AWS Backup
#Step 5: Create the Maintenance Window and register the scan-only task
1aws ssm create-maintenance-window
2 --name "prod-webtier-patch-scan"
3 --schedule "cron(0 2 ? * SUN *)"
4 --duration 3 --cutoff 1 --allow-unassociated-targets
5
6aws ssm register-target-with-maintenance-window
7 --window-id mw-0123456789abcdef0
8 --resource-type INSTANCE
9 --targets "Key=tag:Patch Group,Values=production-webtier"
10
11aws ssm register-task-with-maintenance-window
12 --window-id mw-0123456789abcdef0
13 --task-arn "AWS-RunPatchBaseline"
14 --task-type RUN_COMMAND
15 --targets "Key=WindowTargetIds,Values=[wt-0abc123]"
16 --task-invocation-parameters '{"RunCommand":{"Parameters":{"Operation":["Scan"]}}}'
17 --max-concurrency "50%" --max-errors "10%"On the scheduled Sunday 02:00 run, the task executes with Operation: Scan, which reports compliance without installing anything. Record the maintenance window execution ID and per-instance status in the change ticket. Run scan-only for at least one full cycle before you register an Install-mode task against the same fleet — a single successful scan is not enough evidence to promote.
#Staged rollout and change control
Do not run Install mode against the whole fleet on day one, however tidy Step 5 looked. Split the Patch Group into a small canary subset — two or three non-critical instances — and register a separate task pointing only at that subset. Watch that canary through a full patch and reboot cycle, confirm application health checks pass afterwards, and only then widen the target to the remaining fleet in a second change record. This staged approach is what separates a controlled rollout from a fleet-wide outage caused by one unexpected package interaction.
| Stage | Scope | Approval needed |
|---|---|---|
| Scan-only | Full Patch Group | Standard change, informational |
| Canary install | 2–3 non-critical instances | CAB approval, low-risk window |
| Full install | Remaining Patch Group members | CAB approval referencing successful canary evidence |
#Step 6: Promote to Install mode with change approval
Once scan history is clean and CAB has approved a change window, register a second task with "Operation":["Install"] against a separate, explicitly scheduled maintenance window. Never overwrite the scan-only window in place — keep the two as distinct artefacts so scan history and install history remain independently auditable.
#Monitoring and alerting
A compliance report sitting in a ticket is only useful on the day it was produced. Build a CloudWatch alarm against the SSM compliance summary metric, or schedule a recurring compliance export via EventBridge, so drift is caught before the next scheduled scan rather than a week after a new critical CVE lands. Route the alarm to an SNS topic that the named owner from your prerequisites checklist actually monitors — an alarm nobody reads is functionally the same as no alarm.
#Verification
1aws ssm describe-instance-patch-states
2 --instance-ids i-0123456789abcdef0
3 --query "InstancePatchStates[*].[InstanceId,PatchGroup,MissingCount,InstalledCount,FailedCount]"
4
5aws ssm list-compliance-items
6 --resource-ids i-0123456789abcdef0
7 --resource-types "ManagedInstance"
8 --filters Key=ComplianceType,Values=PatchThe deliverable for this task is a compliance report export — CSV from the Fleet Manager Compliance dashboard, or the CLI output above — attached to the change record, showing FailedCount: 0 across the target fleet, along with the maintenance window execution IDs for both the scan and install runs. Retain the pre-patch snapshot IDs alongside this evidence for at least one full patch cycle before you consider deleting them.
#Failure Modes or Common Traps
| Symptom | Likely cause | Fix |
|---|---|---|
Instance absent from describe-instance-information | Missing or wrong instance profile; SSM Agent stopped | Attach a role with AmazonSSMManagedInstanceCore; restart the agent service |
| Maintenance window task status shows a timeout | Window duration too short for patch volume, or agent unreachable mid-run | Increase --duration; verify the VPC endpoint or NAT path is stable |
| Instance stuck non-compliant after a successful install | Patch required a reboot that never happened | Set RebootOption: RebootIfNeeded in the task invocation parameters |
| Baseline shows zero approved patches | Approval rule classification filter too narrow, or the approval delay has not yet elapsed | Widen the classification list or reduce the approval delay for the next cycle |
| Hybrid or on-prem host never registers | Missing SSM activation code or ID | Run aws ssm create-activation and re-register the agent |
| Compliance report and Step 1 instance count no longer match | New instances launched without the Patch Group tag applied at boot | Bake the tag into the launch template or Auto Scaling group tag propagation settings |
#Rollback
If an Install-mode run breaks an instance — failed boot, service crash-loop, dependency conflict — do not attempt to hand-fix packages under pressure. Restore from the pre-patch snapshot taken in Step 4:
1aws ec2 create-volume --availability-zone eu-west-2a
2 --snapshot-id snap-0abc123def456ghi7 --volume-type gp3
3
4# stop the instance, detach the damaged root volume, attach the restored volume as root, start the instance
5aws ec2 stop-instances --instance-ids i-0123456789abcdef0
6aws ec2 detach-volume --volume-id vol-damaged0000000
7aws ec2 attach-volume --volume-id vol-restored00000
8 --instance-id i-0123456789abcdef0 --device /dev/xvda
9aws ec2 start-instances --instance-ids i-0123456789abcdef0To halt further scheduled patching immediately while you investigate, disable the maintenance window rather than deleting it, so its history is preserved:
1aws ssm update-maintenance-window
2 --window-id mw-0123456789abcdef0 --no-enabledLog the rollback action, the failing patch or package name, and the snapshot ID used for restore in the incident record. Only re-enable the window once the failing patch has been excluded through a baseline rejection rule, or once a vendor fix is confirmed and tested against the canary subset again.
#Hands-on task and completion check
Task: in a non-production sandbox account, tag two test instances with Patch Group = sandbox-test, create a custom baseline restricted to Security classification with a zero-day approval delay, run a scan-only maintenance window task, then promote to an Install run, and produce the compliance CSV showing both instances at zero missing critical patches.
Completion check: you should be able to state, without re-reading this document, the exact CLI command to list instance patch states, the exact tag key Patch Manager routes on, and the correct operational sequence — snapshot, scan, approve, install — before you are trusted with a production maintenance window.
#Operational Summary
Patch Manager only ever behaves as well as your tagging discipline and your instance profile hygiene — nearly every failed rollout traces back to one of those two, not to the service itself. Keep scan-only and install-mode maintenance windows as separate, independently auditable artefacts. Stage the rollout through a canary subset before touching the full fleet. Always snapshot before an install run against production, and never promote a baseline to Install mode without a signed-off change record and a rollback path you have actually rehearsed rather than assumed would work. Review the baseline approval rule syntax and the maintenance window task registration options in the official AWS Systems Manager documentation before your first production install run.
#Learning Objectives
These objectives extend the standard operating procedure above into diagnostic competence: being able to read maintenance window execution evidence, not just trigger it. By the end of working through this material you should be able to independently interpret the artefacts a Patch Manager run produces, rather than relying solely on the summary compliance export.
Focus areas are: distinguishing a scan-only task result from an install-mode result at the execution-status level; tracing a single instance's patch outcome back to the underlying Run Command invocation that produced it; and recognising, from status codes alone, whether a failure originated in connectivity, IAM, or the patch content itself.
- Differentiate maintenance window execution statuses (Success, TimedOut, Failed, Cancelled) and what each implies for change-record evidence
- Drill from a window execution ID down to a per-instance command invocation to inspect raw patch output
- Identify, from invocation output alone, whether a failure is a connectivity gap, an IAM/role gap, or a genuine patch conflict
- Explain why read-only diagnostic queries carry no rollback obligation, while any command that cancels or mutates a running task does
#Worked Example
After the Step 5 scan-only run completes, do not rely on the compliance dashboard alone to close the change ticket; pull the execution record directly so the ticket contains primary evidence rather than a derived summary. Run: aws ssm describe-maintenance-window-executions –window-id mw-0123456789abcdef0 –max-results 5. Validation: the response should contain a WindowExecutionId with Status shown as SUCCESS for the Sunday run; if Status shows TIMED_OUT, the failure mode table's duration guidance applies before you investigate further.
Take that WindowExecutionId and drill into the task level: aws ssm describe-maintenance-window-execution-task-invocations –window-execution-id wei-0abc123def456 –task-id task-0abc123. Validation: each target instance should list its own Status field independently — a single instance showing FAILED inside an otherwise SUCCESS window execution is common and does not invalidate the rest of the run, but it must be captured in the ticket as a named exception rather than folded silently into the aggregate pass.
For that one failing instance, retrieve the underlying command output: aws ssm get-command-invocation –command-id cmd-0abc123def456 –instance-id i-0123456789abcdef0. Validation: StandardOutputContent and StandardErrorContent together should indicate whether the failure was a repository timeout, a missing dependency, or an agent-side permission error — this is the evidence you attach to the change ticket exception, not a guess written from memory.
These three commands are read-only queries against SSM's execution history; they mutate no resource state, so no rollback is required or applicable when running them. If you instead need to stop a task invocation that is visibly hung mid-run, aws ssm cancel-command –command-id cmd-0abc123def456 is the correct action; validation is a subsequent get-command-invocation showing Status: Cancelled, and there is no rollback path to resume a cancelled invocation — the only recovery is re-registering the task and triggering a fresh run once the underlying blocker (usually connectivity or a stuck package manager lock) is confirmed cleared.
#Practice Exercise
In the same non-production sandbox used for the hands-on task, deliberately induce a task-invocation failure so you can practise diagnosis rather than only the happy path. Temporarily remove the sandbox instance's route to the SSM VPC interface endpoint (or NAT path) by detaching the relevant security group rule, then trigger the scan-only maintenance window task manually against that instance.
Expected result: the instance drops out of describe-instance-information as Online, and the subsequent task invocation for that instance reports a connectivity-related failure rather than a patch-content failure. Use describe-maintenance-window-execution-task-invocations and get-command-invocation, as in the worked example, to confirm the failure signature matches a network gap rather than an IAM or baseline problem — this distinction is the exercise's actual objective.
Rollback for this exercise: re-attach the security group rule you removed, confirm the instance returns to PingStatus: Online in describe-instance-information, then re-run the scan-only task against that single instance and confirm Status: SUCCESS in the task invocation record before reverting it back into the shared Patch Group target list. Do not leave the sandbox instance detached from its endpoint route after the exercise concludes.
Completion check for this exercise: you should be able to state, from the invocation output alone and without consulting the failure-mode table again, whether a given failed task invocation is a connectivity issue, an IAM/role issue, or a genuine patch conflict, and name the exact CLI command sequence used to reach that conclusion.
Comments
Add a thoughtful note on Automated EC2 Patch Compliance with AWS Systems Manager. Comments are checked for spam and held for moderation before appearing.
Related articles
Cloud Infrastructure and Operations
Automating EC2 Start/Stop Schedules to Cut Non-Prod Costs
A hands-on runbook for junior engineers to build automated EC2 start/stop schedules with EventBridge and Lambda, cutting non-production AWS costs.
Cloud Infrastructure and Operations
Enforcing AWS Auto Scaling Health Checks for Resilient EC2
A hands-on runbook for converting EC2-only Auto Scaling health checks to ELB-aware checks, with tuning, failover proof, evidence and rollback steps.
Enterprise IT Management
Designing a Verifiable IT Management Workflow with Microsoft 365
A bounded Microsoft 365 Conditional Access workflow: staged rollout through report-only evaluation and pilot enforcement, explicit validation gates, and a rehearsed, non-destructive rollback path.
Systems Engineering
Failure-Aware PowerShell Architecture for a Bounded IT Toolkit
A bounded, failure-aware PowerShell pattern for IT Toolkit-style service workflows: snapshot before change, ShouldProcess-gated actions, transcript evidence and a verified rollback path.
Discover more
Graduate Learning
Learn More About KBY
About KBY
Learn about our mission, editorial standards, and commitment to trusted engineering knowledge.
Why Trust KBY
Explore the processes and policies that ensure our publications are accurate, useful, and responsible.
Newsletter
Get our latest editorial publications, research and practical insights sent directly to your inbox.
Was this useful?
Build practical engineering skills.
Receive new lessons, learning paths, practical exercises and early-career guidance.