Locking Down Production RDS Backups with AWS Backup Vault Lock
A hands-on runbook for locking production RDS backups with AWS Backup Vault Lock and cross-region copy, covering rollback limits and audit evidence.
Priya NairIn this lesson
Table of Contents
Table of contents
Before you begin
- AWS IAM permissions and role management
- AWS CLI configuration and usage
- RDS instance operations basics
Track this tutorial
Choose your current status and tick each safety check as you complete it. Sign in to sync progress between devices.
Current status
Tutorial stages
- Step 1: Create backup vaults with dedicated KMS keys in both regions
- Step 2: Build a backup plan with a cross-region copy action
- Step 3: Validate the service role and run a manual backup job
- Step 4: Apply Vault Lock, governance mode first
0 of 4 stages complete
Before you apply the change
Confirm these production-safety controls during the tutorial.
#Operational requirement
You are going to get handed a ticket that reads something like: “harden backupsbackup:DeleteRecoveryPoint and backup:DeleteBackupVault. There was nothing left to restore from. That single fact is the reason this procedure exists.
AWS Backup Vault Lock makes recovery points immutable, and in compliance mode it makes them deletion-proof against every principal in the account, including root. Combine it with cross-region copy and you also satisfy the availability requirement: a regional outage, or an attacker who only has access to one region, cannot destroy your only copy of production data. This is a realistic task for a newly hired administrator responsible for RDS resilience. It is also unforgiving. Get the lock mode wrong and there is no undo button.
Do not point this procedure at production on your first attempt. Build it against a throwaway vault first.
#Prerequisites and required permissions
You need an existing production RDS instance, a change management process that will genuinely approve an irreversible action, and the following permissions attached to your working role, ideally through a scoped permission set rather than administrator access: backup:CreateBackupVault, backup:CreateBackupPlan, backup:CreateBackupSelection, backup:StartBackupJob, backup:PutBackupVaultLockConfiguration, backup:DeleteBackupVaultLockConfiguration, kms:CreateKey, kms:CreateGrant, and iam:PassRole scoped to the AWS Backup service role.
You also need a second AWS region already reachable from the account, with no Organizations SCP blocking it, MFA enforced on the identity performing the lock operation, and a Change Advisory Board reference number written into the ticket before you start. Vault Lock in compliance mode is one of the few AWS actions where “we’ll fix it later” simply does not exist as an option. The approval trail matters more here than on almost any other change you will make this year.
#Step 1: Create backup vaults with dedicated KMS keys in both regions
Create a customer-managed KMS key per region rather than accepting the AWS managed backup key. You need explicit key policy control later for the cross-region copy step.
1aws kms create-key --region us-east-1 --description "prod-rds-backup-primary"
2aws backup create-backup-vault
3 --backup-vault-name prod-rds-vault
4 --encryption-key-arn arn:aws:kms:us-east-1:111122223333:key/PRIMARY-KEY-ID
5 --region us-east-1
6
7aws kms create-key --region us-west-2 --description "prod-rds-backup-secondary"
8aws backup create-backup-vault
9 --backup-vault-name prod-rds-vault-dr
10 --encryption-key-arn arn:aws:kms:us-west-2:111122223333:key/SECONDARY-KEY-ID
11 --region us-west-2Attach resources by tag, not by explicit ARN, so future RDS instances inherit protection automatically without a second change ticket:
1aws backup create-backup-selection
2 --backup-plan-id <plan-id-from-previous-output>
3 --backup-selection '{"SelectionName":"prod-rds-tagged","IamRoleArn":"arn:aws:iam::111122223333:role/AWSBackupDefaultServiceRole","ListOfTags":[{"ConditionType":"STRINGEQUALS","ConditionKey":"backup:tier","ConditionValue":"production-rds"}]}'
4 --region us-east-1Expect a SelectionId returned with no error. Record the plan ID and selection ID in the ticket. Tag the RDS instance with backup:tier=production-rds before moving on, and give it at least fifteen minutes before the next evaluation window. Miss that window and the first job silently skips the resource, with no error surfaced anywhere.
#Step 3: Validate the service role and run a manual backup job
Confirm the AWS Backup service role carries both managed policies, not just one:
aws iam list-attached-role-policies --role-name AWSBackupDefaultServiceRoleYou need AWSBackupServiceRolePolicyForBackup for the backup jobs themselves and AWSBackupServiceRolePolicyForRestores for the restore test in Step 4. Force a manual run instead of waiting on the cron schedule:
1aws backup start-backup-job
2 --backup-vault-name prod-rds-vault
3 --resource-arn arn:aws:rds:us-east-1:111122223333:db:prod-orders-db
4 --iam-role-arn arn:aws:iam::111122223333:role/AWSBackupDefaultServiceRole
5 --region us-east-1Run the command, then poll aws backup describe-backup-job --backup-job-id <id> until State reads COMPLETED. Separately confirm a matching copy job with aws backup list-copy-jobs --by-state COMPLETED --region us-east-1, showing a DestinationRecoveryPointArn in us-west-2. If the copy job sits in RUNNING past the completion window, or fails on an access-denied condition, stop. Fix the destination KMS key policy before you go anywhere near Vault Lock.
#Step 4: Apply Vault Lock, governance mode first
| Mode | Can be removed? | When to use it |
|---|---|---|
| Governance (within grace period) | Yes, by an authorised principal | Testing, initial rollout, staging environments |
| Compliance (after grace period expires) | No, not by anyone, including root | Final production state once tested and CAB-approved |
Vault Lock uses a single configuration call. During the grace period you set with --changeable-for-days, it behaves like governance mode and an authorised principal can still remove it. Once that grace period elapses, the lock converts automatically and permanently into compliance mode. There is no warning prompt for this transition.
1aws backup put-backup-vault-lock-configuration
2 --backup-vault-name prod-rds-vault
3 --min-retention-days 35
4 --max-retention-days 100
5 --changeable-for-days 3
6 --region us-east-1Expect no error, and a locked configuration block in the next describe-backup-vault call. Capture the full describe output with a timestamp and the calendar date the grace period expires, and put both in the ticket. Do not extend the grace period past your CAB-approved test window. If restore testing is not finished before the grace period ends, delete and reapply the lock rather than letting it convert unreviewed.
#Verification
Before closing the change ticket, run all of the following and attach the outputs:
1aws backup describe-backup-vault --backup-vault-name prod-rds-vault --region us-east-1
2aws backup list-backup-jobs --by-resource-arn arn:aws:rds:us-east-1:111122223333:db:prod-orders-db
3aws backup list-copy-jobs --by-state COMPLETED --region us-east-1Then perform an actual restore into an isolated non-production VPC. Never into the production subnet. Record the restored instance identifier and the restoration duration. This is the real completion check. If you cannot produce a working restored database from a locked recovery point, the control does not meet the requirement yet, no matter what the console shows.
#Common Traps
| Symptom | Likely cause | Fix |
|---|---|---|
| Copy job fails on an access-denied condition against the destination key | Destination region KMS key policy does not grant the backup service principal kms:CreateGrant and kms:Decrypt | Update the destination key policy to include the backup service principal and the source account |
| Restore test fails on an access-denied condition | AWSBackupServiceRolePolicyForRestores is not attached | Attach the managed policy and retry the restore |
| First scheduled job silently excludes the RDS instance | Tag applied after the plan’s evaluation window, with nothing flagged | Re-tag, wait a full evaluation cycle, confirm via list-backup-jobs |
| Plan update rejected on a retention change | MaxRetentionDays set lower than the age of existing recovery points | Raise the max retention value, or let existing points expire before tightening it |
| Vault becomes immutable earlier than planned | Grace period expired before testing finished | None available. This is exactly what Rollback below covers |
#Rollback
If you are still inside the grace period, rollback is straightforward:
1aws backup delete-backup-vault-lock-configuration
2 --backup-vault-name prod-rds-vault
3 --region us-east-1Confirm with describe-backup-vault that the lock configuration block is gone. To remove the cross-region copy action without touching the lock, edit the plan JSON to drop the CopyActions array and run aws backup update-backup-plan with the revised document. To decommission the plan while keeping recovery points intact, delete the backup selection first, then the plan. The vault and its stored recovery points are unaffected by plan deletion.
If the grace period has already expired and the vault has converted to compliance mode, say so plainly to your manager rather than trying to work around it quietly. There is no API rollback at that point. Recovery points can only be deleted once each one individually passes its MinRetentionDays. The remaining avenue is an AWS Support exception request, which is not guaranteed and is reserved for extraordinary documented circumstances, not routine engineering mistakes. This is exactly why the grace period exists, and exactly why you test in governance mode against a disposable vault before ever pointing this procedure at production.
#Operational Summary
The deliverable for this task is a locked, cross-region-replicated backup vault protecting a named production RDS instance, backed by documented evidence: the vault ARNs and encryption key ARNs, the backup plan and selection IDs, at least one completed backup job and one completed cross-region copy job, the lock configuration output with its grace period expiry date, and a successful restore into an isolated non-production environment with a recorded duration. File all of it against the CAB ticket number before you close the change.
Vault Lock is one of the few controls in AWS where the safe default is deliberate slowness. Test in governance mode first. Prove the restore actually works. Get sign-off from your change board. Only then let the grace period run out into compliance mode. Treat the irreversibility as the feature it is meant to be, not as a risk you talked yourself into, and only after you have personally verified the restore path end to end.
#Learning Objectives
By the end of working through this procedure, you should be able to distinguish governance mode from compliance mode in practical terms, not just definitionally: you should be able to state, from your own testing, exactly what breaks and what does not when a lock configuration is deleted versus when the grace period has already lapsed.
You should also be able to explain the cross-region copy dependency chain: source vault key, destination vault key, the AWS Backup service role, and the specific KMS actions each side needs, because a gap anywhere in that chain produces the access-denied failure mode described in the Common Traps table, and diagnosing it quickly depends on understanding the chain rather than pattern-matching the error text.
A further objective is being able to produce audit-ready evidence without being told what to collect each time: vault ARNs, key ARNs, plan and selection IDs, job IDs in both COMPLETED and RUNNING states, and a restore duration, assembled as a matter of habit rather than as a checklist read off a ticket template.
- Differentiate governance-mode and compliance-mode behaviour through direct testing rather than documentation alone
- Trace the KMS permission chain required for a cross-region backup copy to succeed
- Assemble CAB-ready evidence for a locked, replicated backup configuration without prompting
#Worked Example
The article instructs you to fix the destination KMS key policy when a copy job fails on access-denied, without showing the policy statement itself. In practice, the destination key policy needs a statement granting the AWS Backup service principal, backup.amazonaws.com, the actions kms:CreateGrant, kms:DescribeKey, and kms:Decrypt, scoped with a condition requiring kms:ViaService to match the destination region's backup service endpoint. Without the kms:ViaService condition, the grant is broader than necessary and will likely be flagged in a security review even after the copy job succeeds.
Before editing the key policy, retrieve and save the existing document with aws kms get-key-policy –key-id SECONDARY-KEY-ID –policy-name default –region us-west-2, redirecting the output to a local file. This is your rollback artefact. Only after that file exists should you apply the revised policy with aws kms put-key-policy –key-id SECONDARY-KEY-ID –policy-name default –policy file://revised-policy.json –region us-west-2.
Validate by re-running the failed copy job manually rather than waiting for the next scheduled window: aws backup start-copy-job with the same source recovery point ARN and destination vault ARN used previously. Poll aws backup describe-copy-job until State reads COMPLETED. If it still fails, the fault is elsewhere in the chain, most likely the service role's managed policies checked in Step 3, not the key policy.
Rollback, if the new policy statement causes an unrelated permission regression elsewhere, is to reapply the saved original document with the same put-key-policy call, then re-validate with get-key-policy that the reverted JSON matches the saved file byte for byte before considering the rollback complete.
#Practice Exercise
Using a disposable, non-production vault only, build a monitoring layer the base procedure does not include: an EventBridge rule matching AWS Backup job state changes, filtered to state DENIED and FAILED, with an SNS topic as the target. This closes the gap where, as the article notes, a missed tagging window fails silently with no error surfaced anywhere.
Create the SNS topic first, subscribe your own email address, and confirm the subscription before wiring anything else, since an unconfirmed subscription will silently drop notifications in exactly the same way the article warns about elsewhere. Then create the EventBridge rule with an event pattern matching source aws.backup and detail-type Backup Job State Change, targeting the SNS topic ARN.
Validate by manually triggering a failure condition in your disposable environment only, such as pointing a backup selection at a resource ARN with an intentionally revoked IAM role, and confirming the SNS notification arrives within a few minutes. Then restore the correct role and confirm the next manual job succeeds without alerting.
Rollback is deleting the EventBridge rule, deleting the SNS subscription and topic, and confirming with aws events list-rules and aws sns list-topics that neither resource remains, before discarding the disposable vault itself with aws backup delete-backup-vault once it holds no recovery points.
Evidence trail
Sources and verification
Primary documentation and external technical references used in this article.
Comments
Add a thoughtful note on Locking Down Production RDS Backups with AWS Backup Vault Lock. Comments are checked for spam and held for moderation before appearing.
Related articles
Cloud Infrastructure and Operations
Enforcing AWS Auto Scaling Health Checks for Resilient EC2
A hands-on runbook for converting EC2-only Auto Scaling health checks to ELB-aware checks, with tuning, failover proof, evidence and rollback steps.
DevOps & Automation
Engineering a Bounded GitHub Actions Deployment Workflow
A scoped GitHub Actions deployment pipeline design covering job architecture, OIDC security, validation evidence, failure modes and a tested rollback path.
Systems Engineering
Building a Bounded PowerShell Validation Workflow for The IT Toolkit
A pattern for wrapping an IT Toolkit PowerShell task in pre-flight checks, verified backups, explicit validation and a tested rollback path, so success and failure are both observable rather than assumed.
Discover more
Graduate Learning
Learn More About KBY
About KBY
Learn about our mission, editorial standards, and commitment to trusted engineering knowledge.
Why Trust KBY
Explore the processes and policies that ensure our publications are accurate, useful, and responsible.
Newsletter
Get our latest editorial publications, research and practical insights sent directly to your inbox.
Was this useful?
Build practical engineering skills.
Receive new lessons, learning paths, practical exercises and early-career guidance.