KBY Technologies Logo
runbooks/core-infrastructure/grafana-dashboard-data-source-timeout.md
HIGH SEVERITYCore InfrastructureGrafana

Grafana Dashboard Data Source Timeout

Severity
HIGH
Target Time
30 minutes to isolate Grafana, proxy, query or backend timeout
DomainCore Infrastructure
Verified
Overview

Diagnose a Grafana dashboard data-source timeout by separating panel-query cost, Grafana data proxy limits, reverse-proxy deadlines, network reachability and backend saturation before raising any timeout.

Share

01 // Diagnose

Symptom

Incident signalWhat responders observe

Grafana panels show timeout, context deadline exceeded, query error or no data after waiting, while Save & test may pass or fail. One panel, one data source, one tenant or the entire instance can be affected. A longer dashboard range or higher-cardinality query may reproduce the problem even when basic health checks succeed.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. Record Grafana version, data-source plugin/version, data-source UID, dashboard/panel, time range, query, failure timestamp and exact message. Redact credentials and tenant headers.

  2. Use Query Inspector to capture the evaluated request, timing, response size and error.

  3. Re-run with a short recent range and a narrower query.

  4. Run Save & test or the data-source health endpoint with a restricted service account, then test the backend from the Grafana server network path.

  5. Correlate Grafana server/data-proxy logs, reverse-proxy logs and backend query logs using the same timestamp or trace ID.

  6. Compare elapsed time with the data-source query timeout, Grafana dataproxy timeout, TLS handshake timeout, reverse-proxy timeout and backend limit; the lowest deadline wins.

Root Cause Analysis

Causal chainWhy the incident occurred

  1. The query scanned too much history or cardinality and exceeded the data source or backend query limit.

  2. Grafana could not reach or authenticate to the backend because of DNS, routing, firewall, proxy, TLS or tenant-header configuration.

  3. Grafana's data proxy or an upstream reverse proxy timed out before the backend answered.

  4. Backend saturation, connection-pool exhaustion, locks or resource limits delayed the response.

  5. Provisioned data-source configuration diverged from the UI and was reapplied after restart.

  6. A plugin or Grafana upgrade changed query behaviour or exposed a version-specific defect.

02 // Contain & Prevent

Blast Radius

  • Dashboards and alert evaluations that share the data source can become slow, fail or show stale/no data.

  • Broad retries and longer timeouts can amplify backend load and connection-pool exhaustion.

  • Debug logs and copied Query Inspector payloads can expose queries, labels, tenant identifiers or credentials if handled carelessly.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Alert separately on Grafana request duration/errors and data-source backend saturation.

  • Use bounded dashboard ranges, label filters, aggregation, appropriate minimum interval and recorded query budgets.

  • Keep provisioned data-source configuration version controlled and test it before rollout.

  • Align proxy and backend deadlines intentionally.

  • Increase a timeout only after proving the query should legitimately take longer.

  • Review the Grafana-Prometheus integration and Prometheus cardinality controls for metrics workloads.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Save the dashboard, panel query, data-source provisioning file and relevant Grafana/proxy configuration.

  2. Confirm whether the data source is provisioned; do not edit it in the UI if automation will overwrite it.

  3. Obtain a restricted Grafana service-account token without embedding it in command history or the incident record.

  4. Establish backend latency and query-size baselines before changing timeouts.

  5. Route query, provisioning, reverse-proxy and Grafana configuration changes through their owners with rollback copies.

Execution CommandsCOMMANDS

# Grafana service and recent logs (read-only).
systemctl status grafana-server
journalctl -u grafana-server --since "-15 min" --no-pager

# Optional authenticated health check; keep the token in a protected environment variable.
curl --fail --silent --show-error --max-time 15 -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" "$GRAFANA_URL/api/datasources/uid/$DATASOURCE_UID/health"

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. Re-run the affected panel with the original time range and confirm it completes within the agreed query budget.

  2. Verify Save & test or the health endpoint succeeds and the backend receives the expected query.

  3. Confirm Grafana, proxy and backend timeout/error rates return to baseline for at least 30 consecutive minutes.

  4. Test a second dashboard or alert rule using the same data source to rule out panel-only recovery.

  5. Remove temporary debug logging and restricted tokens, then verify normal log level and alert evaluation.

Rollback Protocol

Safe reversal path

  1. Revert the panel query, provisioning file, Grafana setting or reverse-proxy change from its captured version.

  2. If a higher timeout worsens saturation, restore the prior deadline first and reduce query load.

  3. Revoke any temporary service-account token and reset debug logging to info after evidence collection.

Escalation

Conditions requiring additional ownership

  • The same timeout affects multiple data sources or Grafana instances.

  • Backend saturation, connection exhaustion or query correctness cannot be resolved by the owning team.

  • The health API is unavailable because the deployment has migrated from the documented legacy endpoint.

  • Use the instance Swagger/API specification with the Grafana owner.

  • A plugin/Grafana regression or Grafana Cloud service issue is suspected.

  • Escalate with Grafana/plugin versions, redacted Query Inspector output, trace IDs and correlated logs.

Authoritative Sources

Authoritative · Grafana Labs

Troubleshoot general data source issues

Documents connection, query-timeout, configuration and debug-log investigation.

Authoritative · Grafana Labs

Query inspector in Explore

Documents query timing, raw request/response data and error inspection.

Authoritative · Grafana Labs

Configure Grafana

Defines Grafana data-proxy logging and timeout configuration.

Authoritative · Grafana Labs

Data source HTTP API

Documents the UID-based data-source health endpoint and required authentication.

Grafana Dashboard Data Source Timeout - Incident Runbook | KBY Technologies