AWS Identity Controls Leave Workloads Exposed Until Verified

Symptom

A workload can be reachable or privileged in ways its owners did not intend, even though identity controls appear to be configured.

False assumption

The common assumption is that a single identity control, such as a policy attachment, is sufficient on its own. Whether that holds depends on how every applicable policy layer is evaluated, so confirm the details in the AWS documentation.

Root cause

The root cause for a specific case must be established from observed evidence in the affected account. Version-specific and service-specific behaviour should be confirmed in the AWS documentation before it is stated as fact.

Impact

The impact depends on which identities and workloads are affected. Measure it by listing the principals and resources involved rather than inferring it. Related reading: how a missing sts:ExternalId enables confused deputy attacks.

Diagnosis

Use read-only inspection in an isolated or non-production account. Record the policies, trust relationships and access results that apply to the affected identity, and note the date and tooling used. See also IAM role chaining through Lambda PassRole wildcards.

Correction

No change should be applied until the cause is evidenced. Any correction should be the smallest scoped change, reviewed by a second person and tested in a non-production account first.

Validation

Success is observable: the previously unintended access attempt is denied, intended access still works, and the access logs show both results.

Rollback

Before any change, export the existing policy documents so they can be restored exactly. Stop and restore if intended access breaks or the results differ from expectation.

Prevention

Prerequisites and permissions

Agree the scope of the investigation before running any command. Name the account ID, the Region or Regions, the affected role or user, and the workload that assumes it. Use a dedicated read-only investigation role rather than a personal administrator session, so that every query is attributable in the audit trail.

Minimum read permissions for the investigator

The investigation role needs permission to read, not to write. A practical starting set of actions is listed below. Check each one against the current AWS documentation for the services in scope, and add further read actions only when a specific query fails for want of them.

• iam:GetRole, iam:ListAttachedRolePolicies, iam:ListRolePolicies and iam:GetRolePolicy for identity policies and trust relationships.
• iam:GetPolicy and iam:GetPolicyVersion for managed policy documents.
• iam:GetRole returns the permissions boundary reference where one is set.
• iam:SimulatePrincipalPolicy for policy simulation.
• organizations:ListPolicies and organizations:DescribePolicy, held in the management or delegated administrator account, for service control policies.
• cloudtrail:LookupEvents and read access to the log bucket for historical evidence.

Confirm that the investigator cannot call any Put, Attach, Delete or Update action. A failed write attempt in a test session is useful evidence that the role is genuinely read-only.

Approvals needed before changes

Obtain a named change approver, a named second reviewer and, where the workload is shared, sign-off from the workload owner. Record all three in the change ticket before the first write call.

Implementation detail: collecting the evidence

Create a working directory named with the date and ticket reference, and write every output to a file. Run the commands with an explicit profile so the correct account is queried.

Capture the identity layer

aws sts get-caller-identity --profile investigator
aws iam get-role --role-name WORKLOAD_ROLE --profile investigator > role.json
aws iam list-attached-role-policies --role-name WORKLOAD_ROLE --profile investigator > attached.json
aws iam list-role-policies --role-name WORKLOAD_ROLE --profile investigator > inline-names.json

For each managed policy, fetch the default version and keep the version ID alongside the document. For each inline policy, fetch it with aws iam get-role-policy. The AssumeRolePolicyDocument inside role.json is the trust policy; read its Principal and Condition elements line by line, and note any principal that is an entire account rather than a specific role.

Capture the other policy layers

An identity policy is only one input to an access decision. Note whether the role has a permissions boundary, whether the account sits under service control policies, and whether the target resource carries its own resource policy. Export each document and record where it came from. State plainly in your notes which layers you could not read, because an unread layer is an open question and not a clean result.

Test the access question directly

Use the policy simulator for the unintended action and the intended action, against a specific resource ARN rather than a wildcard:

aws iam simulate-principal-policy \
  --policy-source-arn ROLE_ARN \
  --action-names ACTION_UNDER_TEST \
  --resource-arns RESOURCE_ARN \
  --profile investigator

Treat the simulator result as a prediction. Confirm against the documentation which policy types it evaluates, and then confirm the actual behaviour with a real attempt in the non-production account.

Expected evidence

A complete evidence pack lets a reviewer reach the same conclusion without access to the live account. It should contain:

• The caller identity output showing which account and principal ran the queries.
• Every policy document, trust policy and boundary, with version IDs and retrieval timestamps.
• The simulator output for both the unintended and intended actions.
• CloudTrail events showing the real calls, including the principal, source, time and outcome.
• A short written statement linking the observed access to the specific statement that grants it.

Compute a checksum for each file, for example sha256sum *.json > evidence.sha256, and attach the checksum file to the ticket. That makes later tampering or accidental edits detectable.

Monitoring

Monitoring should tell you that access has drifted before a user reports it. Build it from events already recorded rather than from assumptions.

Events worth alerting on

• Changes to the role’s trust policy, policy attachments, inline policies and boundary.
• New principals appearing in successful AssumeRole events for the affected role.
• Use of the role from a source or at a time outside its known pattern.
• Access-denied results for intended actions, which indicate a correction has gone too far.

Route these through an EventBridge rule or a CloudWatch Logs metric filter on the CloudTrail log group, whichever your account already uses. Test each alert by making a harmless change in the non-production account and confirming that the notification arrives, and record how long it took.

Scheduled review

Re-run the evidence collection on a fixed schedule and compare the output with the previous pack using diff. An unexplained difference in a policy document or trust relationship is a reason to open an investigation.

Realistic failure symptoms

These are the patterns that suggest identity controls are present but not effective. Each is a prompt to investigate, not a diagnosis.

• A principal that should be refused succeeds, and the policy you expected to block it is attached but not the one that decided the outcome.
• The simulator says denied while a real call succeeds, or the reverse. This points to a layer that one method did not evaluate.
• A role works from one account and fails from another with no matching difference in the identity policy, which points to trust or resource-side conditions.
• After a correction, a scheduled job or deployment pipeline starts failing with an access-denied message. Read the exact message from the log and do not paraphrase it in the ticket.
• Calls succeed intermittently because several roles or sessions with different permissions share the same workload.

Change-control records

Every write action needs a paper trail that a stranger could follow. Keep these items in the change ticket:

1. The evidence pack and its checksum file.
2. The exact proposed change as a policy diff, not a description of one.
3. The names of the author, the second reviewer and the approver, with timestamps.
4. The non-production test result for both the denied and the permitted action.
5. The planned change window and the person on call during it.
6. The rollback artefacts and the location where they are stored.
7. The post-change verification output and the final decision to close or reopen.

Do not edit the original evidence after the change. Add new files for the post-change state so the before and after positions remain comparable.

Escalation thresholds

Define in advance when a human with authority must take over. Suggested triggers, to be adjusted to your own risk appetite, are:

• Evidence that an unintended principal has already used the access against production data or resources. Treat this as a security incident and hand over to the incident process.
• Any finding that depends on version-specific or service-specific behaviour you cannot confirm from the documentation.
• A proposed change that touches a shared role, a service control policy or a trust policy used by more than one workload.
• Intended access failing during testing and not recovering after the first rollback.
• Any gap in the logs covering the period under investigation, because the scope of exposure then cannot be established.

When escalating, send the ticket reference, the evidence pack location, what you have established, what you have not, and the next decision required.

Safe rollback actions

Prepare the rollback before the change, not after a failure. The exported policy documents are the source of truth for restoration.

Before the change

Store the original documents in a location with restricted write access and versioning enabled. For a customer managed policy, note the current default version ID. Managed policies keep a limited number of versions, so check the documented limit and delete an old non-default version if necessary to leave room for the new one.

Restoring

To restore a managed policy, set the previous version as the default with aws iam set-default-policy-version, naming the saved version ID. To restore an inline policy or trust policy, reapply the exported document with aws iam put-role-policy or aws iam update-assume-role-policy. Afterwards, fetch the live document again and compare it with the export using diff; the two should be identical.

After restoring

Re-run the intended-access test to confirm service is back, then record in the ticket that the original exposure has returned and remains open. A rollback restores availability, not safety, so reschedule the correction with a narrower scope and a fresh review rather than leaving the exposure in place unrecorded.

Review identity policy changes in a non-production account, keep a dated record of the evidence, and re-confirm behaviour against current AWS documentation. Escalate to a human reviewer for any version-specific claim.