Skip to main content

Expiring Untagged ECR Images Deletes the Digest an ECS Service Still Runs, and New Tasks Fail to Pull

Medium severityAmazon Web Services About 20 minutes to fix

Overview

New ECS tasks fail with CannotPullContainerError after an ECR rule expired untagged images. The service was pinned to a digest that lost its tag on the last push of latest.

At a glance

Symptom
New Amazon ECS tasks stop with CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref, while the tasks that were…
Likely cause
ECS starts every task in a service from one image digest, and the cleanup rule deleted that digest.
Impact
A service pinned to an expired digest cannot start a new task until it gets a new deployment.
Verification signal
The fix holds when new tasks start from an image ECR still has, under a tag the policy will not expire.
Safe correction
Deploy a uniquely tagged image first and change the lifecycle policy second, so the service always has an image it can start.
Rollback or recovery
Each step below can be reversed; an expired image can only be pushed again from a build you still have.

Symptom

New Amazon ECS tasks stop with CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref, while the tasks that were already running stay healthy. If the Amazon ECR lifecycle rules changed recently, check first for a rule that expires untagged images. The service is pinned to an image digest, that digest lost its tag when the pipeline pushed a newer image under the same tag, and the rule then deleted it. The fix is to deploy a revision that names a unique tag or digest that exists, then rewrite the lifecycle policy so it never selects an image a service can still start. Do not start by disabling version consistency or by forcing a redeploy of :latest.

The change behind it is routine: someone adds a cleanup rule close to AWS’s own example that keeps one untagged image and expires the rest, and nothing fails for hours or days. Then a scale-out or a replaced task needs one more copy of the container, and it cannot start. Running tasks do not pull again, so the service looks partly healthy.

Why the cleanup rule looks safe

An untagged image looks like an orphan: aws ecr list-images shows it with a digest and no tags. In a repository with mutable tags, though, every push of :latest, :prod or :main moves that tag to the new image. The image the service was deployed from keeps its digest, but unless it carries another tag it is now untagged, and an untagged rule selects it.

Root Cause

ECS starts every task in a service from one image digest, and the cleanup rule deleted that digest. By default ECS resolves the tag in the task definition to a digest: for a service, the first task the scheduler starts during a deployment establishes the digest, and ECS then uses that digest to start every other desired task and for future service updates, so every task in the service runs an identical image. AWS’s CannotPullContainer troubleshooting page states the consequence plainly: ECS forces image version stability, and if the original image is no longer available you get this error.

On the registry side, a lifecycle rule can select untagged images by count or by days since push, the expire action deletes them, and ECR expires matching images within 24 hours of them meeting the criteria. Nothing in that evaluation asks ECS whether a service still runs a digest.

Impact

A service pinned to an expired digest cannot start a new task until it gets a new deployment. That bites during a scale-out under load or when a failed task is replaced. If the repository replicates deletions to another Region or account, ECR replicates the lifecycle deletion there too, so a standby Region can lose the same image. The expire action deletes the image, so the only way back is to push it again from a build you still have.

Two shortcuts that make it worse

The first is to set versionConsistency to disabled on the container. ECS then stops resolving the tag to a digest and starts tasks from the original image URI, so new tasks start again. But each task now pulls whatever the tag points to at the moment it starts. Tasks in one service can run different builds, and anything pushed to that tag goes live on the next scale-out without a deployment.

The second is to force a new deployment of the same task definition. It works, because a new deployment establishes updated digests for new tasks, but it ships whatever image the tag points to now, which may be a build that never went through the checks the running release did. Use it only once you know the current image under the tag is the one you want. Deleting the lifecycle policy outright stops further deletions, but it cannot bring the expired image back and it leaves every old build in the registry still pullable; old builds carry their own risk, covered in why scan-on-push leaves older images unscanned.

How to confirm the pinned digest is gone

These checks are read-only. Replace prod, web and the ARNs with your own.

  1. Find the task definition new tasks are started from. The deployment with status PRIMARY is the most recent one.
    aws ecs describe-services --cluster prod --services web \
      --query "services[0].deployments[].{status:status,taskDefinition:taskDefinition,running:runningCount,desired:desiredCount}"
  2. Read why recent tasks stopped, and which image and digest they used. Run the same describe-tasks against a running task to see the digest the service is pinned to.
    aws ecs list-tasks --cluster prod --service-name web --desired-status STOPPED
    aws ecs describe-tasks --cluster prod --tasks <task-arn> \
      --query "tasks[].{reason:stoppedReason,containers:containers[].{image:image,digest:imageDigest}}"
  3. Ask ECR for that digest. ImageNotFoundException confirms the image is gone. If it comes back with no imageTags, it is next in line for any untagged rule.
    aws ecr describe-images --repository-name web \
      --image-ids imageDigest=sha256:<digest>
  4. Read the rule with aws ecr get-lifecycle-policy --repository-name web. In CloudTrail, filter on the event name PolicyExecutionEvent, which ECR records when a lifecycle rule expires images, to see when it ran. Searching CloudTrail Event history by event name works the same way here.

Correction

Deploy a uniquely tagged image first and change the lifecycle policy second, so the service always has an image it can start. Before you begin, you need push rights to the repository, permission to register task definitions and update the service, and the build of the release that should be running.

  1. Save what you are about to change: the current lifecycle policy text and the current task definition.
    aws ecr get-lifecycle-policy --repository-name web \
      --query lifecyclePolicyText --output text > lifecycle-before.json
    aws ecs describe-task-definition --task-definition web:42 > taskdef-42.json
  2. Push the release you want under a unique tag, such as a version number or commit ID, not under :latest. AWS’s own guidance for this error is to change the task definition from :latest to a specific version.
  3. Register a new task definition revision whose image is that tag, in the form repository-url/image:tag, or a digest in the form repository-url/image@digest. Then deploy it:
    aws ecs update-service --cluster prod --service web --task-definition web:43

    Stop if new tasks fail again with the same error, and check the tag exists with describe-images before you retry.

  4. Write the new lifecycle policy and preview it before applying it. Under unique tags, deployed images keep their tags, so the untagged rule only reaches builds nothing deploys, and the tagged rule keeps enough releases to roll back to.
    {
      "rules": [
        { "rulePriority": 1, "description": "Keep the last 20 releases",
          "selection": { "tagStatus": "tagged", "tagPatternList": ["v*"],
            "countType": "imageCountMoreThan", "countNumber": 20 },
          "action": { "type": "expire" } },
        { "rulePriority": 2, "description": "Expire untagged builds after 14 days",
          "selection": { "tagStatus": "untagged", "countType": "sinceImagePushed",
            "countUnit": "days", "countNumber": 14 },
          "action": { "type": "expire" } }
      ]
    }
    aws ecr start-lifecycle-policy-preview --repository-name web \
      --lifecycle-policy-text file://policy.json
    aws ecr get-lifecycle-policy-preview --repository-name web

    Wait for status to read COMPLETE, then compare every digest in previewResults with the digests your services run. If any matches, change the rule and preview again. Only then apply it with aws ecr put-lifecycle-policy --repository-name web --lifecycle-policy-text file://policy.json.

Validation

The fix holds when new tasks start from an image ECR still has, under a tag the policy will not expire.

  • describe-services shows the new revision on the PRIMARY deployment, with runningCount equal to desiredCount.
  • describe-tasks on a new task shows an imageDigest that describe-images returns with your unique tag in imageTags.
  • A fresh lifecycle preview lists none of the digests your services use.

Rollback

Each step below can be reversed; an expired image can only be pushed again from a build you still have.

  • Service: roll back with update-service to the previous revision that names a unique tag. Rolling back to the old :latest revision does not bring back the old image: the new deployment resolves the tag again and pins whatever it points to now.
  • Lifecycle policy: restore the saved text with aws ecr put-lifecycle-policy --repository-name web --lifecycle-policy-text file://lifecycle-before.json. Restoring the policy does not restore images it already expired.
  • Tag immutability, if you turn it on below: set it back with aws ecr put-image-tag-mutability --repository-name web --image-tag-mutability MUTABLE.

Stop it happening on the next release

Deploy from unique tags and make them immutable, so a release tag can never be moved to a different image. ECR can keep one tag such as latest movable while every other tag is locked; pushing an existing locked tag then fails with ImageTagAlreadyExistsException.

aws ecr put-image-tag-mutability --repository-name web \
  --image-tag-mutability IMMUTABLE_WITH_EXCLUSION \
  --image-tag-mutability-exclusion-filters filterType=WILDCARD,filter=latest

Preview every lifecycle change against the digests your services run, and remember that a lifecycle policy only acts in the Region where you create it unless deletions are replicated. Be careful with sinceImagePulled rules: ECR updates the last recorded pull time on pulls (at least once every 24 hours), but a stable service that has not started a task recently has not pulled either, so its image can look idle while it is still the only one the service can start.

3

Apply the safer control

Before you change production

Confirm the affected scope, export the current configuration, and test the replacement control in a non-production environment first.

Fix commands and configuration

CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref
:latest
aws ecr list-images
4

When to escalate

Validation and rollback are covered in the sections above.

Escalate

Escalate when the blast radius is uncertain, the control cannot be tested safely, or remediation requires an outage or security exception.

The KBY Brief

Fixed? Get the next one before it hits you: The KBY Brief

Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.

About this page

Credited to

Dates

Published

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

After remediation

Further reading stays below the corrective workflow and is selected by platform, category and shared technical keywords.

Discover more

Connected KBY resources