Adding a Second User-Assigned Managed Identity to an Azure VM Breaks Apps That Never Named Their Identity
Overview
A VM gained a second user-assigned managed identity and an application lost access. IMDS needs the identity named once there are two, and the app never named it.
At a glance
- Symptom
- An application on an Azure virtual machine that has read from Key Vault or Storage for months stops getting tokens.
- Likely cause
- The application's token request names no identity, and Microsoft documents that a VM with more than one user-assigned identity needs one named.
- Impact
- Every process on the VM that asks for a token without naming an identity fails the next time it needs a new token: the application, scheduled scripts that…
- Verification signal
- The fix holds when every workload on the VM gets a token for its own identity and nothing extra was granted along the way.
- Safe correction
- Name the identity in the application before you change anything on the VM.
- Rollback or recovery
- Application setting: remove AZURE_CLIENT_ID or the code option and restart.
Symptom
An application on an Azure virtual machine that has read from Key Vault or Storage for months stops getting tokens. The change that day was routine: someone attached a second user-assigned managed identity to the VM so another workload could use it. The first identity and its role assignments are untouched. The application never said which identity it wanted, and with one user-assigned identity it didn’t have to. With two, the Instance Metadata Service (IMDS) has to be told which one to use, so the token request fails. Fix the application first by giving it the client ID of the identity that holds its roles. Don’t turn on the system-assigned identity, and don’t go back to a client secret.
This applies to a VM with no system-assigned identity that went from one user-assigned identity to two or more. A VM that already has a system-assigned identity behaves differently, as covered below.
Why attaching an identity looks safe
Microsoft supports this setup. A resource can have a system-assigned identity and one or more user-assigned identities, and Microsoft’s best-practice guidance describes sharing one user-assigned identity across many resources so fewer role assignments are needed. Assigning an identity to a VM changes no role assignments and detaches nothing, so in a change review it reads as additive and low risk. The person making the change usually owns the new workload, not the application that breaks.
Root cause
The application’s token request names no identity, and Microsoft documents that a VM with more than one user-assigned identity needs one named. Code on the VM gets a managed identity token from the IMDS endpoint http://169.254.169.254/metadata/identity/oauth2/token. Microsoft documents three optional query parameters that pick the identity: client_id, object_id and msi_res_id. Each is marked as required if the VM has multiple user-assigned managed identities. A request that names none of them worked while there was only one identity to choose. Attach a second, and the same request no longer says which identity to use.
The Azure Identity libraries pass whatever identity they were configured with. In Python, DefaultAzureCredential takes managed_identity_client_id, which defaults to the AZURE_CLIENT_ID environment variable; if neither is set, it uses a system-assigned identity. In .NET, AZURE_CLIENT_ID, if defined, is the default for ManagedIdentityClientId. An application with neither configured sends no identity at all.
IMDS reports request errors with a 4xx status and a JSON body with error and error_description. Its documentation calls 4xx errors design-time errors and says not to retry them, so retry logic will not get the application working again.
Impact
Every process on the VM that asks for a token without naming an identity fails the next time it needs a new token: the application, scheduled scripts that call IMDS with curl or Invoke-RestMethod, and any agent built the same way. The new workload usually works, because whoever set it up named its identity, so the change looks fine to the person who made it.
Three tempting fixes that make it worse
The first is to turn on the VM’s system-assigned identity. The Python library documents that, with no client ID, it uses a system-assigned identity, so token requests succeed again. But the token now belongs to a new identity with none of the application’s roles, so the failure moves to the Key Vault or Storage call. The next step under pressure is to grant that identity the same roles. Two identities then hold the same access, and Microsoft notes that role assignments are not deleted automatically when an identity is deleted.
The second is to fall back to a service principal secret in AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET. DefaultAzureCredential tries a service principal from environment variables before managed identity, so this works. It also brings back the secret that managed identities exist to remove, and you now have to store and rotate it. The same trade appears when a storage client goes back to account keys, covered in fixing Storage 403 errors after Shared Key access is disabled.
The third is to remove the new identity. That restores the old behaviour and breaks the workload it was added for, and the next person to attach an identity will hit the same failure.
How to confirm the diagnosis
These checks change nothing. Replace rg-app, app01 and the IDs with your own.
- List the VM’s identities. You are looking for a type with no
SystemAssignedand two or more entries underuserAssignedIdentities.az vm identity show --name app01 --resource-group rg-app - Get each identity’s client ID and principal ID from its resource ID.
az identity show --ids <identity-resource-id> \ --query "{clientId:clientId, principalId:principalId}" - Find the identity that holds the application’s roles. Without
--all, only subscription-scope assignments are listed.az role assignment list --assignee <principal-id> --all --output table - On the VM, request a token twice: once as the application does, and once with that identity’s client ID. If the first fails and the second returns an
access_token, the diagnosis is confirmed. Don’t paste the token into a ticket.curl -s -H Metadata:true \ 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' curl -s -H Metadata:true \ 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F&client_id=<client-id>'
Correction
Name the identity in the application before you change anything on the VM. That way the VM never has an identity layout the application can’t use. You need access to the application’s configuration. For the identity changes, you also need Virtual Machine Contributor, plus Managed Identity Operator to assign a user-assigned identity.
- Save the current state:
az vm identity show --name app01 --resource-group rg-app > vm-identity-before.json, and a copy of the application’s configuration or service definition. - Pin the application to the identity that holds its roles. Set
AZURE_CLIENT_IDto that identity’s client ID in the application’s environment, or pass it in code asmanaged_identity_client_id(Python) orManagedIdentityClientId(.NET). Don’t addAZURE_TENANT_IDorAZURE_CLIENT_SECRET. Restart the application. If token errors continue, check that the client ID matches an identity listed byaz vm identity showbefore you try anything else. - Do the same for every other process on the VM that requests tokens. Scripts that call IMDS directly need
client_id=in the query string. - If you removed the second identity to get running again, assign it back only after steps 2 and 3:
az vm identity assign -g rg-app -n app01 --identities <identity-resource-id>If anything on the VM starts failing to get a token, remove it again with
az vm identity remove -g rg-app -n app01 --identities <identity-name>and find the process that still sends no identity.
Validation
The fix holds when every workload on the VM gets a token for its own identity and nothing extra was granted along the way.
- The application reads from Key Vault or Storage again, with no token errors in its log.
- The IMDS request with
client_idreturns a token, andaz vm identity showstill lists every identity the VM’s workloads need. - No system-assigned identity or client secret was added on the way. If
az role assignment liston an identity added during the incident shows copies of the application’s roles, remove them once the pinned application has run cleanly, so only one identity holds that access.
If the pinned request gets a token but the resource still answers 403, the problem is authorisation, not identity selection. The Owner role on an RBAC Key Vault returning Forbidden is one case to check next.
Rollback
- Application setting: remove
AZURE_CLIENT_IDor the code option and restart. This is only safe while the VM has a single user-assigned identity. - Identity assignment:
az vm identity remove -g rg-app -n app01 --identities <identity-name>removes an identity you added, andaz vm identity assignputs back one you removed, using the IDs saved invm-identity-before.json.
Stop it happening on the next change
Configure the client ID in every application that uses a user-assigned identity, even when the VM has only one. The setting changes nothing today, and it means the next identity anyone attaches can’t break the application. Before you approve a change that attaches an identity to a VM, ask the owners of everything already running there whether their code names its identity. The two token requests above show it without changing anything.
Apply the safer control
Before you change production
Confirm the affected scope, export the current configuration, and test the replacement control in a non-production environment first.
Fix commands and configuration
http://169.254.169.254/metadata/identity/oauth2/tokenclient_idobject_idWhen to escalate
Validation and rollback are covered in the sections above.
Escalate
Escalate when the blast radius is uncertain, the control cannot be tested safely, or remediation requires an outage or security exception.
Fixed? Get the next one before it hits you: The KBY Brief
Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.
About this page
Credited to
Dates
Published
Corrections
No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.
Was this useful?
Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.