Configuring AWS Budgets and Cost Anomaly Detection for Production

#Operational requirement

You will get this ticket in your first month on the job: finance has flagged an unexpected AWS invoice and wants alerts set up so it never happens again. It happens because someone spun up an oversized RDS instance for a load test and forgot to tear it down, or because a misconfigured Auto Scaling group scaled to maximum capacity over a bank holiday weekend with nobody watching. AWS does not stop you spending money. Left unmonitored, a single production account can burn through a six-figure sum in one billing cycle without a single alert firing.

The consequence of skipping this control is not abstract. Finance escalates a surprise invoice to the CFO, the CFO escalates to your director, and your director asks why Cloud Operations had no visibility. A missed threshold alert on a production account is treated as an operational failure, not a minor oversight, and it will be reviewed as one.

Your job on this ticket is to configure an AWS Budget with actual and forecasted thresholds, enable AWS Cost Anomaly Detection for pattern-based spend spikes that fixed thresholds miss, wire both into an SNS topic that fans out to email and a Slack or Teams channel, and lock the budget down with an IAM guardrail so nobody quietly deletes it during a quick fix. Treat this as a standing control, not a one-off setup task.

#Prerequisites and required permissions

Confirm you hold an IAM identity with the AWS managed policy AWSBudgetsActionsWithAWSResourceControlAccess, or a custom policy granting budgets:*, ce:*, sns:CreateTopic, sns:SetTopicAttributes, and sns:Subscribe. If the account sits under AWS Organizations with consolidated billing, ask your lead whether the budget should be created at the member-account level or centrally via the management account. The two approaches produce very different blast radii for alerting, and getting this wrong means either missing account-level noise or drowning the whole organisation in a single account’s alerts.

You will also need the following in place before you start:

• AWS CLI v2 configured with a profile scoped to the target account, not the management account, unless you are deliberately building an org-wide monitor.
• A resource tagging convention already agreed, minimally an Environment tag with values such as Production or Staging. Untagged spend cannot be filtered cleanly and your alerts will fire on noise or, worse, stay silent while real spend is excluded.
• An existing Slack or Microsoft Teams incoming webhook, or authorisation to create one, plus a target SNS topic that a Lambda function or AWS Chatbot integration can subscribe to.
• A change ticket reference number. Every step below should be logged against it, because this control will be audited later.

Confirm your CLI identity before touching anything:

1aws sts get-caller-identity --profile prod-costops

Expected result: the command returns with no output and exit code 0. Evidence to capture: run aws budgets describe-budgets --account-id 123456789012 immediately afterwards and save the JSON showing prod-monthly-cap with the correct limit. Checkpoint: confirm the budget appears in the Billing console under Budgets before moving on. A malformed CostFilters key can fail without a loud error, and the console view is your independent check.

#Step 3: Enable Cost Anomaly Detection for pattern-based spikes

Fixed thresholds catch slow creep. They do not catch a sudden three-hour spike from a runaway Lambda recursion loop that resolves itself before the monthly threshold trips. That gap is what Cost Anomaly Detection closes.

1{ "MonitorName": "prod-linked-account-monitor", "MonitorType": "DIMENSIONAL", "MonitorDimension": "LINKED_ACCOUNT" }

Capture the returned MonitorArn from the response, then create the subscription. Note that the older Threshold field on anomaly subscriptions is deprecated; use ThresholdExpression instead, or the API will reject the request.

1{
2  "SubscriptionName": "prod-anomaly-sns",
3  "Frequency": "DAILY",
4  "MonitorArnList": ["arn:aws:ce::123456789012:anomalymonitor/xxxxxxxx-xxxx"],
5  "Subscribers": [{ "Address": "arn:aws:sns:eu-west-1:123456789012:budget-alerts", "Type": "SNS" }],
6  "ThresholdExpression": "{"And":[{"Dimensions":{"Key":"ANOMALY_TOTAL_IMPACT_ABSOLUTE","Values":["200"],"MatchOptions":["GREATER_THAN_OR_EQUAL"]}}]}"
7}

Expected result: a SubscriptionArn returned in the response. Evidence: save both ARNs, monitor and subscription, into the ticket. You will need them for rollback and for the periodic review described later.

#Step 4: Wire the SNS topic to Slack or Teams

Action: grant the Budgets and Cost Explorer services permission to publish to your SNS topic, then subscribe an email address as a fallback and a Lambda function or AWS Chatbot integration for the Slack channel.

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Sid": "AllowBudgetsAndAnomalyPublish",
6      "Effect": "Allow",
7      "Principal": { "Service": ["budgets.amazonaws.com", "costalerts.amazonaws.com"] },
8      "Action": "SNS:Publish",
9      "Resource": "arn:aws:sns:eu-west-1:123456789012:budget-alerts"
10    }
11  ]
12}

Expected result: a confirmation email arrives and must be actively confirmed; the subscription stays in PendingConfirmation state until then. Evidence: a CLI listing from aws sns list-subscriptions-by-topic showing Confirmed. Checkpoint: send a test publish before relying on this in production.

1aws sns publish --topic-arn arn:aws:sns:eu-west-1:123456789012:budget-alerts --message "Test alert: ignore"

#Step 5: Lock the budget against silent deletion

Attach a deny policy so only an identity tagged as a FinOps admin can delete or modify the budget or the anomaly monitors. This is the step most junior engineers skip because it feels like an afterthought. It is not. Without it, anyone with budgets permissions can remove the control during an unrelated cleanup and nobody will notice until the next surprise invoice.

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Sid": "DenyBudgetTampering",
6      "Effect": "Deny",
7      "Action": ["budgets:DeleteBudget", "budgets:ModifyBudget", "ce:DeleteAnomalyMonitor", "ce:DeleteAnomalySubscription"],
8      "Resource": "*",
9      "Condition": { "StringNotEquals": { "aws:PrincipalTag/Role": "FinOpsAdmin" } }
10    }
11  ]
12}

Attach this to the relevant permission set or group. Evidence: the IAM policy version ID and the group ARN it was attached to, both logged in the ticket.

#Ongoing monitoring and change control

This is not a set-and-forget control. Put a recurring calendar reminder, or better, an automated report, to review the budget threshold and anomaly sensitivity monthly. Spend patterns shift as workloads grow, and a threshold set correctly at launch becomes either too tight or dangerously loose within a couple of quarters.

Any change to the budget limit, the anomaly threshold, or the SNS subscribers must go through the same change ticket process used for the original setup, including a review by whoever holds the FinOps admin role. Do not adjust the threshold informally because a false alert was annoying during an on-call shift; raise a ticket, document the reasoning, and let the review process catch a threshold that was widened too far.

Watch for tag drift. New services and accounts get provisioned without the agreed tagging convention applied, silently excluding them from the filter. A quarterly audit against aws ce get-tags across all linked accounts catches this before it becomes a gap in coverage.

#Hands-on task

Using a sandbox account, repeat Steps 2 and 3 with a budget limit of 100 USD and an anomaly threshold of 20 USD. Trigger a deliberate breach by launching and leaving running a small EC2 instance tagged Environment=Production for several hours, then confirm the alert arrives in your test Slack channel within the expected daily anomaly detection cycle. Terminate the instance once confirmed, and log the result in a practice ticket exactly as you would for a production change.

#Verification

• aws budgets describe-budgets --account-id 123456789012 shows the budget with the correct limit and filters.
• aws ce get-anomaly-monitors and aws ce get-anomaly-subscriptions both return the expected resources in an enabled state.
• CloudTrail Event History filtered on CreateBudget, CreateAnomalyMonitor, and CreateAnomalySubscription shows the actions attributed to your IAM identity with a timestamp matching the change ticket.
• A test publish from Step 4, or a genuine breach, produced a visible Slack message and a confirmed email subscription.

#Failure Modes

#Rollback

If the change needs to be reversed, for example during an incident where the alert itself is misfiring and paging the on-call team incorrectly, reverse it in this order:

1aws budgets delete-budget --account-id 123456789012 --budget-name prod-monthly-cap
2
3aws ce delete-anomaly-subscription --subscription-arn arn:aws:ce::123456789012:anomalysubscription/xxxxxxxx
4
5aws ce delete-anomaly-monitor --monitor-arn arn:aws:ce::123456789012:anomalymonitor/xxxxxxxx
6
7aws sns unsubscribe --subscription-arn arn:aws:sns:eu-west-1:123456789012:budget-alerts:xxxxxxxx
8
9aws sns delete-topic --topic-arn arn:aws:sns:eu-west-1:123456789012:budget-alerts

Detach the IAM deny policy last, once the underlying resources are already gone, to avoid a window where nothing is monitored and nothing is locked. Record the rollback with a CloudTrail export attached to the change ticket, and note the reason for reversal so the next attempt does not repeat the same misconfiguration.

#Operational Summary

A junior engineer who completes this task correctly leaves behind a budget that fires on both actual and forecasted spend, an anomaly detector that catches pattern-based spikes fixed thresholds miss, an SNS fan-out confirmed to reach both email and chat, and an IAM guardrail preventing quiet deletion. The evidence trail is the describe-budgets output, the anomaly monitor and subscription ARNs, the confirmed SNS subscription state, and the CloudTrail record of who created what and when.

Completion check: if you can produce all four pieces of evidence on demand during a review, the control is operating, not just configured. For the underlying service behaviour, consult the AWS documentation on managing costs with budgets and the AWS documentation on getting started with Cost Anomaly Detection. Cost governance is not a one-off ticket; it is a control you will be asked to prove works every time finance asks how the team caught the last overspend before it became a headline invoice.