Incident Overview
Northwind Retail (a fictional organisation) runs its payments-api service through a shared CI/CD platform. At 09:42 on a Tuesday, an engineer merges a small validation fix to the main branch.
The pipeline builds the service, runs its test suite, and reports a green run: build succeeded, tests passed, and the promotion stage marks the new artifact as deployed to the production cluster.Fifteen minutes later, the on-call platform engineer notices something odd. The CI/CD dashboard still shows the deployment as fully successful and up to date with the latest commit.
However, a subset of customer requests to payments-api are returning an old validation error that the merged fix was supposed to remove. Other requests, routed to different replicas, behave correctly.
There is no dashboard alert, no failed health check, and no obvious rollback trigger u2014 only an inconsistent pattern in production behaviour that contradicts the pipeline's own success signal.
Investigation Options
Review the available operational moves and select the best immediate action.
Roll back the payments-api deployment to the last verified-good artifact tag and freeze the build cache pending investigation.
Force a full cache invalidation and rebuild payments-api from a clean cache before taking any deployment action.
Restart the deployment gate service and re-trigger promotion of the currently reported artifact.
Pause all CI/CD promotions across every service in the organisation until the discrepancy is fully understood.