Prometheus · Alert-rule review
Prometheus Alert Review Checklist
A checklist for validating Prometheus alert meaning, query correctness, duration, labels, routing, runbooks and failure behaviour.
Choose the right control
Use this template when
- Adding or materially changing a Prometheus alerting rule.
- Reviewing noisy, stale or unactionable alerts before renewing them.
Use another workflow when
- Designing raw instrumentation or dashboard-only queries.
- The receiver has no defined action and the signal is intended only for exploration.
Before you start
Collect these inputs before review so unchecked controls become evidence-backed decisions rather than placeholders.
Representative metric history including normal and failure periods
The deployed rule group and evaluation interval
An owned receiver, escalation path and runbook
How to use this template
1
Test rules against representative historical data.
2
Review the receiving team's ability to act on the signal.
3
Delete or redesign alerts that do not lead to a bounded decision.
Copyable template
# Prometheus alert review ## Signal - [ ] Alert represents customer impact, exhaustion or a necessary operator decision - [ ] PromQL expression is correct across missing, stale and reset series - [ ] Label matchers select only the intended services and environments - [ ] Threshold is supported by capacity, SLO or failure evidence ## Timing - [ ] Evaluation interval and for-duration match the failure timescale - [ ] Alert resolves when the actionable condition clears - [ ] Short data gaps do not create misleading recovery or firing loops - [ ] Expected deployment and maintenance behaviour is accounted for ## Routing and context - [ ] Ownership labels route to a staffed receiver - [ ] Summary identifies the affected service and condition - [ ] Annotations link to a current, permission-accessible runbook - [ ] Grouping and inhibition avoid notification floods ## Validation - [ ] promtool validates the rule file - [ ] Unit tests cover firing, non-firing and missing-data cases - [ ] A test notification reaches the intended receiver - [ ] Review date and accountable owner are recorded
Evidence to retain
Review record
- promtool validation and unit-test output
- Historical query screenshots or reproducible evaluation timestamps
- A routed test notification and resolved-state observation
Definition of done
Acceptance criteria
- The expression fires for the intended failure and remains quiet during normal operation.
- Labels route to a staffed owner without unbounded cardinality.
- The alert links to a current action that can change the outcome.
Worked adaptation
Apply it to a real change
A high-error-rate alert pages during every deployment because the denominator briefly disappears. The review uses historical data to distinguish real customer impact from missing-series behaviour.
- Add firing and missing-data fixtures to the rule tests.
- Align the for-duration with the deployment observation window.
- Verify grouping prevents one notification per pod.
Related KBY resources
Engineering signal
Follow Prometheus failures and fixes
A useful digest about twice a week with new error references, tools, integration notes and production lessons. No daily noise.