KBY Technologies Logo
runbooks/server/systemd-service-failed-state-after-configuration-deployment.md
HIGH SEVERITYServersystemd

A systemd Service Enters Failed State After a Configuration Deployment

Severity
HIGH
Target Time
30m
DomainServer
Verified
Overview

Diagnose a failed systemd unit after configuration deployment, restore the last approved configuration, and restart the service with bounded verification.

Share

Destructive Operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

01 // Diagnose

Symptom

Incident signalWhat responders observe

A Linux service managed by systemd is inactive or failed immediately after a configuration deployment, and dependent requests are timing out or being refused.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. Run systemctl is-failed <UNIT> and systemctl status <UNIT> --no-pager --full and record the result, exit status, and recent messages.

  2. Run journalctl -u <UNIT> --since '<INCIDENT_START>' --no-pager and capture the first failure after deployment.

  3. Run systemctl cat <UNIT> to inspect the effective unit file and drop-ins.

  4. Run systemctl show <UNIT> -p FragmentPath -p DropInPaths -p ExecStart -p EnvironmentFiles -p Result -p ExecMainStatus.

  5. Validate the deployed application configuration with its documented read-only validation command before attempting a restart.

  6. Check failed dependencies with systemctl list-dependencies --failed <UNIT>.

Root Cause Analysis

Causal chainWhy the incident occurred
  1. The deployment introduced an invalid application configuration, unit drop-in, environment file, permission, or dependency condition. systemd records the unit as failed after its start command exits unsuccessfully or a start limit is reached; repeated restarts without correcting the cause can preserve or amplify the outage.

02 // Contain & Prevent

Blast Radius

  • The failed service cannot accept new work on the affected host.

  • Dependent units and upstream load balancer health checks may also fail.

  • Restarting before restoring valid configuration can extend the outage or trigger repeated crash loops.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Validate application and unit configuration before deployment and before daemon reload.

  • Deploy configuration atomically with an automatic rollback on failed health checks.

  • Alert on systemd failed units, start-limit hits, and repeated restarts.

  • Keep unit drop-ins and environment files in reviewed configuration management with known-good revisions.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Confirm the current hostname, environment, and <UNIT> are the exact affected host and unit; record systemctl status, systemctl cat, and relevant journal output.

  2. Run sudo -v and confirm the operator is authorized to manage this unit on this host.

  3. Resolve the unit deterministically with systemctl show <UNIT> -p FragmentPath --value, confirm the absolute path belongs to the intended unit, and review every path from systemctl show <UNIT> -p DropInPaths --value.

  4. Identify the deployment change and preserve copies or revision identifiers for both current and last approved configuration.

  5. systemctl provides no true dry-run for daemon-reload, reset-failed, or restart.

  6. Capture SHA-256 hashes of the resolved unit and drop-ins, run systemd-analyze verify plus the application's supported validator as the equivalent preview, then re-check those hashes immediately before daemon-reload; abort if validation fails or any file changed during the review window.

  7. Confirm upstream capacity can tolerate the host remaining out of service during one controlled restart.

  8. Check whether the unit is stateful and obtain the service owner's approval before restarting it.

  9. Restore the last approved configuration through the normal configuration-management or deployment mechanism; do not edit an untracked live file.

  10. Define an abort threshold of one failed restart.

Execution CommandsCOMMANDS

set -e; FRAGMENT=$(systemctl show <UNIT> -p FragmentPath --value); DROPINS=$(systemctl show <UNIT> -p DropInPaths --value); test -n "$FRAGMENT"; CHECKSUM_FILE=$(mktemp); trap 'rm -f "$CHECKSUM_FILE"' EXIT; sha256sum "$FRAGMENT" $DROPINS > "$CHECKSUM_FILE"; sudo systemctl cat <UNIT>; sudo systemd-analyze verify "$FRAGMENT"; <APPLICATION_CONFIG_VALIDATOR>; sha256sum -c "$CHECKSUM_FILE"; sudo systemctl daemon-reload; sudo systemctl reset-failed <UNIT>; sudo systemctl restart <UNIT>; sudo systemctl is-active <UNIT>

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. Confirm systemctl is-active <UNIT> returns active and systemctl is-failed <UNIT> does not return failed.

  2. Review journalctl -u <UNIT> --since '<RECOVERY_START>' --no-pager for new errors.

  3. Exercise the service's documented local health check and one representative read-only request.

  4. Confirm the host has rejoined its load balancer or dependency pool only after health is stable.

  5. Monitor restart count, error rate, latency, and saturation for at least 15 minutes.

Rollback Protocol

Safe reversal path

  1. If the controlled restart fails or the service is unhealthy, stop after the single approved attempt, keep the host out of rotation, restore the captured last approved application and unit configuration through configuration management, run the supported configuration validator, perform one additional owner-approved restart, and escalate if it does not recover.

  2. Do not loop restarts or clear the failed state repeatedly.

Escalation

Conditions requiring additional ownership

  • The last approved configuration also fails validation.

  • The service is stateful and restart safety cannot be established.

  • A dependency, filesystem, credential, kernel, or resource-exhaustion failure remains after configuration rollback.

  • The unit fails the one controlled restart or multiple hosts show the same deployment failure.

Authoritative Sources

Authoritative · systemd

systemctl

Supported unit status, failed-state reset, daemon reload, restart, and activity checks.

Authoritative · systemd

journalctl

Filtering and reading systemd journal records for a specific unit and incident interval.

Authoritative · systemd

systemd-analyze

Verification of unit files and their dependencies before reloading or restarting a unit.

A systemd Service Enters Failed State After a Configuration Deployment - Incident Runbook | KBY Technologies