Incident Command Simulator
A timed decision simulator for practising command, containment and recovery during a multi-region service incident. It focuses on clear roles, reversible actions, evidence preservation and disciplined recovery gates.
Who this Lab is for
Designed for
- Incident commanders and technical leads
- Senior on-call engineers
- Teams rehearsing major-incident procedures
Use it when
- Preparing engineers for incident-command responsibility
- Running individual practice between team game days
- Reviewing command principles after an incident
A complete run, step by step
Select the incident
Choose payment degradation, identity failure or queue saturation.
Make the first command decision
Establish control, assign roles and protect evidence while the situation is uncertain.
Choose containment
Limit blast radius with the most reversible action supported by current evidence.
Set a recovery gate
Require customer and system signals to stabilise before declaring resolution or resuming changes.
What you will need
Prepare the following information before starting. Use measured evidence where possible; defaults are examples and should not be treated as recommendations.
Available scenarios
Connection-pool exhaustion following a client rollout.
Token validation failures spread across regions.
Retry amplification overwhelms downstream consumers.
Minute 2 — first command
Latency rises after a database client rollout.
Choices: Freeze changes, assign roles, preserve telemetry · Restart every database node · Wait for more customer reports
Minute 8 — containment
One region shows connection-pool exhaustion.
Choices: Rollback the client in the affected region · Fail all traffic into the degraded region · Double every pool limit
Minute 18 — recovery gate
Error rate falls but queues remain elevated.
Choices: Verify saturation, drain rate and customer errors · Declare resolved immediately · Resume deployments
What the result tells you
Your report includes
- A command-decision score
- Feedback on containment and recovery discipline
- A saved record across multiple incident variants
How it is determined
Decisions are weighted for command clarity, evidence preservation, reversibility, customer-risk reduction and verification before recovery. Fast but uncontrolled actions score poorly even when they could sometimes work.
The score evaluates three command choices in a simplified scenario and cannot assess team communication or live telemetry.
Model assumptions
- • Reducing concurrent change is appropriate during uncertainty.
- • Targeted rollback is supported by the stated evidence.
- • Recovery requires customer and system signals to stabilise.
Authoritative references
Payment degradation after rollout
Situation
Latency rises after a database client release and one region shows connection-pool exhaustion.
Result
Freezing changes, assigning roles and rolling back only the affected region produces a controlled path to recovery with explicit verification gates.
Use the result with engineering judgement
- The simulator cannot reproduce team communication or incomplete telemetry.
- Real incidents require your approved command structure and escalation paths.
- A high score does not certify someone to command an incident alone.
Questions before you begin
Is this a game day?
It is individual decision practice. Combine it with facilitated team game days for communication, tooling and coordination practice.
Why favour reversible actions?
Early incident hypotheses are uncertain. Reversible containment reduces the chance of making impact worse while evidence develops.
When is an incident resolved?
When customer impact has ended, key system signals are stable and the recovery state has been verified—not simply when one metric improves.
Incident Command is under review
This legacy judgement-based Lab has been retired. Existing saved reports remain available, but new execution is disabled.
Open deterministic utilities