Fixing AWS S3 AccessDenied When an EC2 Instance Role Reads a Bucket

An error such as An error occurred (AccessDenied) when calling the GetObject operation means S3 received a signed request and refused it. The request reached S3 and was authenticated as some identity, so the network path is usually not the problem. The useful question is which identity made the request and which policy decision denied it. Guessing at policies without answering that first is the most common cause of wasted time and of over-broad permissions.

This guide gives an ordered method: confirm the credentials, then the role’s identity policy, then the bucket’s resource policy, then the other controls that can override both. Each step produces evidence that rules out one layer. The guide is written for a non-production account. Production changes need the approvals described near the end.

#Learning Objectives

• Explain how an EC2 instance obtains temporary credentials through an instance profile.
• Distinguish identity policies, resource policies and other controls that can deny S3 access.
• Use read-only commands to identify the calling principal and test a policy decision.
• Apply a least-privilege fix with a recorded rollback.

#Prerequisites

• An isolated or non-production AWS account, and permission to read IAM and S3 configuration there.
• Shell access to a test EC2 instance with the AWS CLI installed. Confirm your CLI version and the current AWS documentation before relying on any output format here.
• A test bucket holding a harmless object. Never use credentials or private production data in an exercise.
• Basic familiarity with JSON
  The KBY Lexicon

  JSON

  JSON (JavaScript Object Notation) is a lightweight, text-based data-interchange format built from key-value pairs and ordered lists, defined by RFC 8259.

  Read Full Context
  policy documents.

#Content

#

The mental model

Terms first. A principal is the identity making a request. An IAM role is an identity with permissions but no long-term password or keys. An instance profile is the container that attaches a role to an EC2 instance. The AWS SDKs and CLI normally fetch temporary credentials for that role from the instance metadata service and refresh them automatically. An identity policy is attached to the role. A resource policy, here a bucket policy, is attached to the bucket. Each S3 request is evaluated against all applicable policies.

The general evaluation logic, which should be confirmed against current IAM documentation, is as follows. A request is denied by default. An explicit Deny in any applicable policy overrides every allow. Within a single account, an allow in either the identity policy or the bucket policy can be enough. Across accounts, both sides generally need to allow. Other layers can also deny: service control policies, permission boundaries, session policies, S3 Block Public Access, VPC endpoint policies and, for encrypted objects, KMS key policies.

#

Why the order of checks matters

The order is chosen so that each check narrows the search and the cheapest checks come first.

#

Step 1: confirm who is calling

Environment variables and shared credential files take precedence over the instance profile in the standard credential lookup order. A stale AWS_ACCESS_KEY_ID left in a shell can therefore make the CLI act as a different identity from the role you are inspecting. Confirm the identity before reading any policy.

1aws sts get-caller-identity

The expected output contains an Arn of the form arn:aws:sts::ACCOUNT_ID:assumed-role/ROLE_NAME/INSTANCE_ID. If it shows an IAM user or a different role, the credentials are the problem. Remove the overriding source rather than editing policies.

#

Step 2: inspect the role’s permissions

1aws iam list-attached-role-policies --role-name ROLE_NAME
2aws iam list-role-policies --role-name ROLE_NAME

These list managed and inline policies. Then read the relevant document. Two S3 details cause frequent confusion. s3:GetObject applies to object ARNs, written arn:aws:s3:::BUCKET/*. s3:ListBucket applies to the bucket ARN, arn:aws:s3:::BUCKET. When a caller lacks s3:ListBucket, S3 generally returns AccessDenied instead of NoSuchKey for a missing object, so a typo in a key can look like a permissions fault.

#

Step 3: inspect the bucket policy

1aws s3api get-bucket-policy --bucket BUCKET_NAME

Look for any statement with "Effect": "Deny" and read its Condition block. Common conditions test the source VPC endpoint, source IP, TLS use or a required encryption setting. A deny keyed to aws:SourceVpce will block an instance whose traffic does not pass through that endpoint, even when the role policy is correct. If the call returns NoSuchBucketPolicy, there is no bucket policy, so a same-account failure points back to the role or another control.

#

Step 4: test the decision without changing anything

1aws iam simulate-principal-policy --policy-source-arn ROLE_ARN --action-names s3:GetObject --resource-arns arn:aws:s3:::BUCKET_NAME/KEY

The simulator evaluates identity-based policies and can include permission boundaries. It does not model every control, such as a bucket policy unless you supply it, or every SCP condition, so treat a result of allowed as evidence about the role only. A result of explicitDeny or implicitDeny identifies which side to fix.

#Examples

#

Worked example: missing object-level allow

Input: the instance runs aws s3api head-object --bucket test-bucket --key reports/a.csv and receives a 403. Step 1 returns the expected assumed-role ARN. The role has a policy allowing s3:ListBucket on arn:aws:s3:::test-bucket only. There is no bucket policy.

Interpretation: credentials are correct and nothing denies explicitly. The role grants listing but not reading, so the implicit default deny applies to GetObject. The fix is a statement allowing s3:GetObject on arn:aws:s3:::test-bucket/reports/*, scoped to the prefix the workload needs and not to every object.

Recovery: before editing, record the current policy version or document so it can be restored. If the change has unwanted effects, reattach the previous version. Re-run the failing request afterwards. Policy propagation can take a short time, so retry once before assuming the change failed.

#

Worked example: explicit deny in the bucket policy

Input: the simulator reports the role as allowed, yet the request fails. The bucket policy contains a Deny with a StringNotEquals condition on aws:SourceVpce.

Interpretation: the role is not at fault. The request does not arrive through the permitted endpoint, and an explicit deny beats the role’s allow. Correct the network path or ask the bucket owner to review the condition. Adding more allows to the role would change nothing.

#Exercises

#

Exercise: trace a denied read in a lab account

Objective: identify the layer that denies a read, using read-only commands.

Setup: a non-production account, a test instance with an instance profile, a test bucket with one harmless object, and a role that lacks s3:GetObject. Prepare the lab so the failure is deliberate and known to you.

1. Run aws sts get-caller-identity on the instance and note the ARN.
2. List the role’s policies and read them.
3. Fetch the bucket policy, or note that none exists.
4. Run the policy simulation for s3:GetObject on the object ARN.
5. Write one sentence naming the denying layer and the evidence for it.

Expected evidence: an assumed-role ARN matching your role, a policy list without a GetObject allow, and an implicitDeny simulation result.

Pass condition: your sentence names the role policy as the cause and cites those outputs.

Stop conditions: stop if the account is not the designated lab account, if the bucket holds real data, or if the caller identity is unexpected. Do not edit anything until you can explain the cause.

Cleanup: this exercise makes no changes. If you tested a fix, restore the saved policy version and unset any credentials exported in the shell.

#Validation Guidance

Success is observable: the original command that failed now succeeds, returning object metadata or content, and an unrelated object outside the permitted prefix still returns AccessDenied. The second check confirms least privilege, since a fix that opens everything also makes the first test pass. Re-run aws sts get-caller-identity afterwards to confirm the same principal was used for both tests. Record the before and after outputs for the change record.

If the request still fails after both policies allow it, check the object’s encryption. Objects encrypted with a customer-managed KMS key require the caller to be permitted to use that key, and the failure may surface as AccessDenied. Also check the object’s ownership settings for cross-account uploads.

#Common Mistakes

• Editing policies before confirming the identity. Symptom: edits change nothing. Cause: an environment variable overrides the instance profile. Diagnosis: get-caller-identity. Correction: remove the override. Recovery: none needed if nothing was edited.
• Using the wrong resource ARN. Symptom: the allow exists but the read fails. Cause: GetObject granted on the bucket ARN without /*. Correction: grant on the object ARN.
• Misreading a missing key as a permissions fault. Symptom: AccessDenied for an absent object. Cause: no s3:ListBucket. Correction: verify the key, then decide whether listing should be granted.
• Fixing with s3:* on *. Symptom: it works. Cause: a quick shortcut. Consequence: far wider access than the workload needs. Correction: scope to the bucket and prefix.

#

Production bridge

In production, a policy change is a change to a security boundary. Obtain approval from the bucket and role owners, make the change through your normal change process, and keep the prior policy version for rollback. If the denial comes from an SCP, permission boundary, endpoint policy or KMS key policy, escalate to the team owning that control and do not attempt to bypass it. Escalate also when the caller identity is unexpected, since that can indicate leaked or misconfigured credentials.

#Key Takeaways

• AccessDenied means S3 authenticated the request and then refused it, so identify the principal first.
• Check in order: credentials, role policy, bucket policy, other controls.
• An explicit deny overrides any allow, and object and bucket actions use different ARNs.
• Validate with the failing request and a negative test, and keep a rollback copy before any edit.

Before closing the work, record the caller ARN, the denying layer, the policy version you can restore and the owner who approved the change. Then repeat the failing request and one out-of-scope request. If either result is unexpected, restore the saved policy version and escalate to the control owner.