Skip to main content

Adding a Second User-Assigned Managed Identity to an Azure VM Breaks Apps That Never Named Their Identity

Medium severityMicrosoft Azure About 20 minutes to fix

Overview

A VM gained a second user-assigned managed identity and an application lost access. IMDS needs the identity named once there are two, and the app never named it.

At a glance

Symptom
An application on an Azure virtual machine that has read from Key Vault or Storage for months stops getting tokens.
Likely cause
The application's token request names no identity, and Microsoft documents that a VM with more than one user-assigned identity needs one named.
Impact
Every process on the VM that asks for a token without naming an identity fails the next time it needs a new token: the application, scheduled scripts that…
Verification signal
The fix holds when every workload on the VM gets a token for its own identity and nothing extra was granted along the way.
Safe correction
Name the identity in the application before you change anything on the VM.
Rollback or recovery
Application setting: remove AZURE_CLIENT_ID or the code option and restart.

Symptom

An application on an Azure virtual machine that has read from Key Vault or Storage for months stops getting tokens. The change that day was routine: someone attached a second user-assigned managed identity to the VM so another workload could use it. The first identity and its role assignments are untouched. The application never said which identity it wanted, and with one user-assigned identity it didn’t have to. With two, the Instance Metadata Service (IMDS) has to be told which one to use, so the token request fails. Fix the application first by giving it the client ID of the identity that holds its roles. Don’t turn on the system-assigned identity, and don’t go back to a client secret.

This applies to a VM with no system-assigned identity that went from one user-assigned identity to two or more. A VM that already has a system-assigned identity behaves differently, as covered below.

Why attaching an identity looks safe

Microsoft supports this setup. A resource can have a system-assigned identity and one or more user-assigned identities, and Microsoft’s best-practice guidance describes sharing one user-assigned identity across many resources so fewer role assignments are needed. Assigning an identity to a VM changes no role assignments and detaches nothing, so in a change review it reads as additive and low risk. The person making the change usually owns the new workload, not the application that breaks.

Root cause

The application’s token request names no identity, and Microsoft documents that a VM with more than one user-assigned identity needs one named. Code on the VM gets a managed identity token from the IMDS endpoint http://169.254.169.254/metadata/identity/oauth2/token. Microsoft documents three optional query parameters that pick the identity: client_id, object_id and msi_res_id. Each is marked as required if the VM has multiple user-assigned managed identities. A request that names none of them worked while there was only one identity to choose. Attach a second, and the same request no longer says which identity to use.

The Azure Identity libraries pass whatever identity they were configured with. In Python, DefaultAzureCredential takes managed_identity_client_id, which defaults to the AZURE_CLIENT_ID environment variable; if neither is set, it uses a system-assigned identity. In .NET, AZURE_CLIENT_ID, if defined, is the default for ManagedIdentityClientId. An application with neither configured sends no identity at all.

IMDS reports request errors with a 4xx status and a JSON body with error and error_description. Its documentation calls 4xx errors design-time errors and says not to retry them, so retry logic will not get the application working again.

Impact

Every process on the VM that asks for a token without naming an identity fails the next time it needs a new token: the application, scheduled scripts that call IMDS with curl or Invoke-RestMethod, and any agent built the same way. The new workload usually works, because whoever set it up named its identity, so the change looks fine to the person who made it.

Three tempting fixes that make it worse

The first is to turn on the VM’s system-assigned identity. The Python library documents that, with no client ID, it uses a system-assigned identity, so token requests succeed again. But the token now belongs to a new identity with none of the application’s roles, so the failure moves to the Key Vault or Storage call. The next step under pressure is to grant that identity the same roles. Two identities then hold the same access, and Microsoft notes that role assignments are not deleted automatically when an identity is deleted.

The second is to fall back to a service principal secret in AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET. DefaultAzureCredential tries a service principal from environment variables before managed identity, so this works. It also brings back the secret that managed identities exist to remove, and you now have to store and rotate it. The same trade appears when a storage client goes back to account keys, covered in fixing Storage 403 errors after Shared Key access is disabled.

The third is to remove the new identity. That restores the old behaviour and breaks the workload it was added for, and the next person to attach an identity will hit the same failure.

How to confirm the diagnosis

These checks change nothing. Replace rg-app, app01 and the IDs with your own.

  1. List the VM’s identities. You are looking for a type with no SystemAssigned and two or more entries under userAssignedIdentities.
    az vm identity show --name app01 --resource-group rg-app
  2. Get each identity’s client ID and principal ID from its resource ID.
    az identity show --ids <identity-resource-id> \
      --query "{clientId:clientId, principalId:principalId}"
  3. Find the identity that holds the application’s roles. Without --all, only subscription-scope assignments are listed.
    az role assignment list --assignee <principal-id> --all --output table
  4. On the VM, request a token twice: once as the application does, and once with that identity’s client ID. If the first fails and the second returns an access_token, the diagnosis is confirmed. Don’t paste the token into a ticket.
    curl -s -H Metadata:true \
      'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F'
    curl -s -H Metadata:true \
      'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F&client_id=<client-id>'

Correction

Name the identity in the application before you change anything on the VM. That way the VM never has an identity layout the application can’t use. You need access to the application’s configuration. For the identity changes, you also need Virtual Machine Contributor, plus Managed Identity Operator to assign a user-assigned identity.

  1. Save the current state: az vm identity show --name app01 --resource-group rg-app > vm-identity-before.json, and a copy of the application’s configuration or service definition.
  2. Pin the application to the identity that holds its roles. Set AZURE_CLIENT_ID to that identity’s client ID in the application’s environment, or pass it in code as managed_identity_client_id (Python) or ManagedIdentityClientId (.NET). Don’t add AZURE_TENANT_ID or AZURE_CLIENT_SECRET. Restart the application. If token errors continue, check that the client ID matches an identity listed by az vm identity show before you try anything else.
  3. Do the same for every other process on the VM that requests tokens. Scripts that call IMDS directly need client_id= in the query string.
  4. If you removed the second identity to get running again, assign it back only after steps 2 and 3:
    az vm identity assign -g rg-app -n app01 --identities <identity-resource-id>

    If anything on the VM starts failing to get a token, remove it again with az vm identity remove -g rg-app -n app01 --identities <identity-name> and find the process that still sends no identity.

Validation

The fix holds when every workload on the VM gets a token for its own identity and nothing extra was granted along the way.

  • The application reads from Key Vault or Storage again, with no token errors in its log.
  • The IMDS request with client_id returns a token, and az vm identity show still lists every identity the VM’s workloads need.
  • No system-assigned identity or client secret was added on the way. If az role assignment list on an identity added during the incident shows copies of the application’s roles, remove them once the pinned application has run cleanly, so only one identity holds that access.

If the pinned request gets a token but the resource still answers 403, the problem is authorisation, not identity selection. The Owner role on an RBAC Key Vault returning Forbidden is one case to check next.

Rollback

  • Application setting: remove AZURE_CLIENT_ID or the code option and restart. This is only safe while the VM has a single user-assigned identity.
  • Identity assignment: az vm identity remove -g rg-app -n app01 --identities <identity-name> removes an identity you added, and az vm identity assign puts back one you removed, using the IDs saved in vm-identity-before.json.

Stop it happening on the next change

Configure the client ID in every application that uses a user-assigned identity, even when the VM has only one. The setting changes nothing today, and it means the next identity anyone attaches can’t break the application. Before you approve a change that attaches an identity to a VM, ask the owners of everything already running there whether their code names its identity. The two token requests above show it without changing anything.

3

Apply the safer control

Before you change production

Confirm the affected scope, export the current configuration, and test the replacement control in a non-production environment first.

Fix commands and configuration

http://169.254.169.254/metadata/identity/oauth2/token
client_id
object_id
4

When to escalate

Validation and rollback are covered in the sections above.

Escalate

Escalate when the blast radius is uncertain, the control cannot be tested safely, or remediation requires an outage or security exception.

The KBY Brief

Fixed? Get the next one before it hits you: The KBY Brief

Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.

About this page

Credited to

Dates

Published

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

After remediation

Further reading stays below the corrective workflow and is selected by platform, category and shared technical keywords.

Discover more

Connected KBY resources