KBY Technologies Logo
runbooks/azure-key-vault-security-incidents/azure-key-vault-secret-reads-403-workload-identity-change.md
HIGH SEVERITYAzure Key Vault Security IncidentsAzure Key Vault

Azure Key Vault Secret Reads Return 403 After a Workload Identity Change

Severity
HIGH
Target Time
45m
DomainAzure Key Vault Security Incidents
Verified
Overview

Determine whether Azure Key Vault 403 responses come from identity, RBAC, firewall, or private DNS changes and restore least-privilege secret access.

Share

Destructive Operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

01 // Diagnose

Symptom

Incident signalWhat responders observe

An Azure workload that previously read a Key Vault secret now receives HTTP 403 Forbidden after its managed identity, deployment slot, or service principal was changed.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. From the workload context, run az account show --query '{tenant:tenantId,subscription:id,user:user}' and record the authenticated tenant and identity.

  2. Run az keyvault show --name <VAULT_NAME> --query '{id:id,rbac:properties.enableRbacAuthorization,publicNetworkAccess:properties.publicNetworkAccess}'.

  3. Run az role assignment list --assignee-object-id <PRINCIPAL_OBJECT_ID> --scope <VAULT_RESOURCE_ID> --include-inherited --all when the vault uses Azure RBAC.

  4. If the vault uses legacy access policies, inspect az keyvault show --name <VAULT_NAME> --query properties.accessPolicies.

  5. Review Key Vault diagnostic logs and the complete 403 inner error to distinguish authorization from firewall or private-endpoint DNS rejection.

  6. Resolve <VAULT_NAME>.vault.azure.net from the workload network and verify the expected public or private endpoint path.

Root Cause Analysis

Causal chainWhy the incident occurred

  1. The redeployed workload is using a different principal object ID or tenant than the identity that holds Key Vault data-plane permission, or the request is reaching a network path rejected by the vault firewall.

  2. Azure control-plane access to the vault resource does not itself grant secret data-plane access.

02 // Contain & Prevent

Blast Radius

  • Applications using the changed identity cannot retrieve required secrets, keys, or certificates.

  • Dependent deployments may fail readiness checks or enter restart loops.

  • A broadly scoped emergency role assignment could expose vault data beyond the affected workload.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Treat managed identity object IDs and Key Vault role assignments as deployment dependencies.

  • Monitor Key Vault 403 responses and retain data-plane diagnostic logs.

  • Test private DNS resolution and secret reads from the workload network during deployment validation.

  • Prefer narrowly scoped Azure RBAC assignments managed through reviewed infrastructure as code.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Confirm the exact principal object ID used by the failing workload and do not substitute an application/client ID.

  2. Confirm the vault uses Azure RBAC; stop and use the approved legacy access-policy procedure if enableRbacAuthorization is false.

  3. Record all current role assignments at the vault scope and the current firewall/private endpoint configuration.

  4. Verify DNS and network path are correct; do not add RBAC permission to solve a firewall rejection.

  5. Confirm the least-privilege role required by the workload and obtain the vault owner's approval.

  6. Ensure the operator can remove the new assignment immediately if the scope or principal is wrong.

Execution CommandsCOMMANDS

az role assignment create --assignee-object-id <PRINCIPAL_OBJECT_ID> --assignee-principal-type ServicePrincipal --role 'Key Vault Secrets User' --scope <VAULT_RESOURCE_ID>
az role assignment list --assignee-object-id <PRINCIPAL_OBJECT_ID> --scope <VAULT_RESOURCE_ID> --include-inherited --all

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. From the workload identity and network path, repeat one read of the required non-test secret and confirm the request succeeds without exposing its value in logs.

  2. Confirm Key Vault logs show the intended principal object ID and a successful data-plane operation.

  3. Verify the assignment scope is the individual vault rather than the subscription or management group.

  4. Confirm an unrelated principal remains denied.

  5. Monitor application health and Key Vault authorization failures for at least 15 minutes.

Rollback Protocol

Safe reversal path

  1. If the assignment targets the wrong principal or grants unintended access, identify the exact new assignment ID from the post-change role-assignment listing and remove it with az role assignment delete --ids <NEW_ASSIGNMENT_ID>.

  2. Re-run the denied-access test for the unintended principal, then restore service through the approved identity or network remediation path.

Escalation

Conditions requiring additional ownership

  • The vault uses legacy access policies or a deny assignment prevents Azure RBAC access.

  • The 403 is caused by private endpoint DNS, firewall policy, or public-network restrictions rather than authorization.

  • The workload's principal cannot be determined unambiguously.

  • Recovery would require subscription-wide access or a role broader than the approved data-plane action.

Authoritative Sources

Authoritative · Microsoft

REST API error codes for Azure Key Vault

Key Vault 403 causes including insufficient permission and firewall restrictions.

Authoritative · Microsoft

Azure role-based access control for Key Vault

Key Vault data-plane RBAC roles, scopes, and role-assignment practices.

Authoritative · Microsoft

Troubleshooting Azure Key Vault access issues

Identity, access policy, RBAC, and diagnostic logging checks for vault access failures.