Skip to main content

Adding a Broker Role in Front of a Deployment Role Caps Its Sessions at One Hour, Whatever MaxSessionDuration Says

Medium severityAmazon Web Services About 20 minutes to fix

Overview

A new broker role in front of a deployment role cut its sessions to one hour. Role chaining ignores MaxSessionDuration, so long deployments now fail.

At a glance

Symptom
A deployment pipeline that ran for hours under one IAM role starts failing after someone puts a broker role in front of it.
Likely cause
Credentials from one role were used to assume another, and AWS caps that second session at one hour.
Impact
Any job behind the broker that runs longer than an hour fails partway through.
Verification signal
The fix holds when the pipeline's longest job completes on a single hop and the role trusts nobody new.
Safe correction
Give the pipeline one hop into DeployRole by trusting the OIDC provider directly, adding the new path before removing the old one.
Rollback or recovery
Every step is reversed by putting back a saved trust policy.

Symptom

A deployment pipeline that ran for hours under one IAM role starts failing after someone puts a broker role in front of it. Before the change, the pipeline signed in through its OIDC provider straight into DeployRole, which has a maximum session duration of four hours, and asked for three. After the change, it signs in to ci-broker first and assumes DeployRole from there. Now the AssumeRole call fails as soon as it asks for more than 3,600 seconds, or, if the duration is removed, long jobs lose their credentials an hour in. Raising DeployRole‘s maximum session duration changes nothing. The second hop is role chaining, and AWS limits a chained role session to one hour whatever the role’s own setting says. The fix is to give the pipeline a single hop into the role it needs, not to raise limits or switch to access keys.

Why a broker role looks safe

A broker role keeps one trust relationship with the CI provider and hands out narrower roles behind it. The deployment role’s permissions don’t change, its trust policy only gains the broker as a principal, and a short test run succeeds. Nothing in the change mentions session length, and the role’s MaxSessionDuration still reads 14,400 seconds in get-role.

Root cause

Credentials from one role were used to assume another, and AWS caps that second session at one hour. The IAM User Guide calls this role chaining and says it limits an AWS Management Console, AWS CLI or API role session to a maximum of one hour, regardless of the maximum session duration configured for the individual roles. The AssumeRole reference adds the consequence: if you assume a role through role chaining and pass a DurationSeconds value greater than one hour, the operation fails.

The first hop isn’t affected. A session from AssumeRoleWithWebIdentity can last from 900 seconds up to the role’s maximum session duration, and that setting can be anything from one to twelve hours. AWS also documents that the one-hour limit doesn’t apply to the initial assumption of a role from user credentials, or to applications on Amazon EC2 instances using instance profiles. So the same deployment role gives a three-hour session when it is the first hop and a one-hour session when it is the second.

Impact

Any job behind the broker that runs longer than an hour fails partway through. That means long infrastructure deployments, large copies and database migrations, which can stop half-applied and need someone to finish or roll them back by hand. Jobs that explicitly request more than an hour fail before they start, which is at least obvious. Jobs that request no duration get the one-hour default, pass every short test, and fail only on the long runs.

Workarounds that cost more than they fix

Raising MaxSessionDuration on DeployRole to twelve hours does nothing for the chained session, because the limit applies regardless of that setting. It does leave every future direct session of that role able to run for twelve hours, which nobody asked for.

Giving the pipeline an IAM user with access keys works, because the limit doesn’t apply when a role is first assumed from user credentials. It also puts long-term credentials back into a CI system. AWS’s IAM best practices say workloads, including managed CI/CD platforms, should use temporary credentials through IAM roles, and keys you create then need rotating and watching, as the trap of a console password reset that leaves access keys active shows.

Granting the broker role the deployment permissions directly removes the second hop, but it turns the broker into the most privileged role in the chain, which is the opposite of what it was added for. AWS’s best practices list least-privilege permissions for this reason.

How to confirm it is role chaining

These checks read configuration or issue short-lived credentials; none of them changes a role. Use your own role names and account ID.

  1. Read the deployment role’s maximum session duration and who it trusts.
    aws iam get-role --role-name DeployRole \
      --query "Role.[MaxSessionDuration,AssumeRolePolicyDocument]"

    A value of 14400 alongside a trust policy that names ci-broker as a principal is the shape of this trap.

  2. In the pipeline, just before it assumes DeployRole, print the identity it is calling from.
    aws sts get-caller-identity

    If Arn is an assumed-role ARN for ci-broker rather than an IAM user, the next AssumeRole call is role chaining.

  3. From the same place, request one hour, then two.
    aws sts assume-role --role-arn arn:aws:iam::111122223333:role/DeployRole \
      --role-session-name chain-check --duration-seconds 3600 \
      --query "Credentials.Expiration"
    aws sts assume-role --role-arn arn:aws:iam::111122223333:role/DeployRole \
      --role-session-name chain-check --duration-seconds 7200 \
      --query "Credentials.Expiration"

    The first returns an expiry an hour away; the second fails, even though 7,200 seconds is well inside the role’s 14,400. Discard the credentials and don’t paste them anywhere.

Correction

Give the pipeline one hop into DeployRole by trusting the OIDC provider directly, adding the new path before removing the old one. You need permission to read and update the role’s trust policy, and the trust statement the broker role uses for your CI provider: its Federated principal, the sts:AssumeRoleWithWebIdentity action and its conditions.

  1. Save the current trust policy so you can put it back exactly.
    aws iam get-role --role-name DeployRole \
      --query "Role.AssumeRolePolicyDocument" --output json > deployrole-trust-before.json
  2. Write deployrole-trust-after.json: the saved document plus one statement copied from ci-broker‘s trust policy, with the same provider and the same conditions. Leave the existing ci-broker statement in place for now. Don’t loosen the conditions while copying them; a wider condition would let more callers into the deployment role.
  3. Apply it.
    aws iam update-assume-role-policy --role-name DeployRole \
      --policy-document file://deployrole-trust-after.json

    If get-role doesn’t show both statements afterwards, or anything that assumed the role before now fails, restore the saved file the same way and stop.

  4. Change one pipeline to assume DeployRole directly with web identity, requesting no more than the role’s MaxSessionDuration, and run its longest job. Move the other pipelines once it completes.
  5. When nothing has used the ci-broker path into DeployRole for a full release cycle, remove that statement with another update-assume-role-policy, saving the document first as in step 1.

If the broker has to stay, for example because it adds checks you rely on, the other honest option is to keep chained sessions under an hour and split long jobs so each stage gets fresh credentials. The AWS CLI documents that a role profile reuses cached temporary credentials until they expire and then refreshes them automatically, which helps a job made of many short commands but not one long call.

Validation

The fix holds when the pipeline’s longest job completes on a single hop and the role trusts nobody new.

  • aws sts get-caller-identity inside the job shows an assumed-role ARN for DeployRole, reached without first becoming ci-broker.
  • The credentials’ Expiration is as far away as the duration the pipeline requested, beyond one hour.
  • aws iam get-role --role-name DeployRole shows the copied OIDC statement with the same conditions as the broker’s, and no wider ones.

Rollback

Every step is reversed by putting back a saved trust policy.

  • Trust policy: aws iam update-assume-role-policy --role-name DeployRole --policy-document file://deployrole-trust-before.json restores the policy exactly as it was.
  • Pipeline: point it back at ci-broker and request no more than 3,600 seconds for the chained session.
  • If you changed MaxSessionDuration while troubleshooting, set it back with aws iam update-role --role-name DeployRole --max-session-duration 14400. AWS notes the new value applies the next time someone assumes the role.

Prevent it on the next role change

Before you put any role in front of another, list the jobs behind it that run longer than an hour, because the chained session can’t outlast that whatever either role says. Check how long the longest stage really runs, not how long it usually runs. Note which roles are reached by chaining in the role’s description, so the next person who reads MaxSessionDuration knows it doesn’t apply on that path. Chaining has other sharp edges too; role chaining through Lambda PassRole wildcards covers the permissions side.

3

Apply the safer control

Before you change production

Confirm the affected scope, export the current configuration, and test the replacement control in a non-production environment first.

Fix commands and configuration

DeployRole
ci-broker
AssumeRole
4

When to escalate

Validation and rollback are covered in the sections above.

Escalate

Escalate when the blast radius is uncertain, the control cannot be tested safely, or remediation requires an outage or security exception.

The KBY Brief

Fixed? Get the next one before it hits you: The KBY Brief

Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.

About this page

Credited to

Dates

Published

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

After remediation

Further reading stays below the corrective workflow and is selected by platform, category and shared technical keywords.

Discover more

Connected KBY resources