Skip to main content

A systemd Service Enters Failed State After a Configuration Deployment

High severity Target recovery 30msystemd

Overview

Diagnose a failed systemd unit after configuration deployment, restore the last approved configuration, and restart the service with bounded verification.

Markdown, one checkbox per step, for pasting into a ticket.

Destructive operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

Diagnose

Symptom

What responders observe.

A Linux service managed by systemd is inactive or failed immediately after a configuration deployment, and dependent requests are timing out or being refused.

Detection

Metrics, logs and confirmation commands.

  1. Run systemctl is-failed <UNIT> and systemctl status <UNIT> --no-pager --full and record the result, exit status, and recent messages.

  2. Run journalctl -u <UNIT> --since '<INCIDENT_START>' --no-pager and capture the first failure after deployment.

  3. Run systemctl cat <UNIT> to inspect the effective unit file and drop-ins.

  4. Run systemctl show <UNIT> -p FragmentPath -p DropInPaths -p ExecStart -p EnvironmentFiles -p Result -p ExecMainStatus.

  5. Validate the deployed application configuration with its documented read-only validation command before attempting a restart.

  6. Check failed dependencies with systemctl list-dependencies --failed <UNIT>.

Commands in these steps

  • systemctl is-failed <UNIT>
  • systemctl status <UNIT> --no-pager --full
  • journalctl -u <UNIT> --since '<INCIDENT_START>' --no-pager
  • systemctl cat <UNIT>
  • systemctl show <UNIT> -p FragmentPath -p DropInPaths -p ExecStart -p EnvironmentFiles -p Result -p ExecMainStatus
  • systemctl list-dependencies --failed <UNIT>

Root cause

Why the incident occurred.

  1. The deployment introduced an invalid application configuration, unit drop-in, environment file, permission, or dependency condition. systemd records the unit as failed after its start command exits unsuccessfully or a start limit is reached; repeated restarts without correcting the cause can preserve or amplify the outage.

Contain and prevent

Blast radius

What else this can affect.

  • The failed service cannot accept new work on the affected host.

  • Dependent units and upstream load balancer health checks may also fail.

  • Restarting before restoring valid configuration can extend the outage or trigger repeated crash loops.

Prevention

Controls and guardrails that stop it recurring.

  • Validate application and unit configuration before deployment and before daemon reload.

  • Deploy configuration atomically with an automatic rollback on failed health checks.

  • Alert on systemd failed units, start-limit hits, and repeated restarts.

  • Keep unit drop-ins and environment files in reviewed configuration management with known-good revisions.

Fix

Pre-flight checks

Check these before you change anything.

  1. Confirm the current hostname, environment, and <UNIT> are the exact affected host and unit; record systemctl status, systemctl cat, and relevant journal output.

  2. Run sudo -v and confirm the operator is authorized to manage this unit on this host.

  3. Resolve the unit deterministically with systemctl show <UNIT> -p FragmentPath --value, confirm the absolute path belongs to the intended unit, and review every path from systemctl show <UNIT> -p DropInPaths --value.

  4. Identify the deployment change and preserve copies or revision identifiers for both current and last approved configuration.

  5. systemctl provides no true dry-run for daemon-reload, reset-failed, or restart.

  6. Capture SHA-256 hashes of the resolved unit and drop-ins, run systemd-analyze verify plus the application's supported validator as the equivalent preview, then re-check those hashes immediately before daemon-reload; abort if validation fails or any file changed during the review window.

  7. Confirm upstream capacity can tolerate the host remaining out of service during one controlled restart.

  8. Check whether the unit is stateful and obtain the service owner's approval before restarting it.

  9. Restore the last approved configuration through the normal configuration-management or deployment mechanism; do not edit an untracked live file.

  10. Define an abort threshold of one failed restart.

Commands in these steps

  • systemctl status
  • systemctl cat
  • sudo -v
  • systemctl show <UNIT> -p FragmentPath --value
  • systemctl show <UNIT> -p DropInPaths --value

Commands

set -e; FRAGMENT=$(systemctl show <UNIT> -p FragmentPath --value); DROPINS=$(systemctl show <UNIT> -p DropInPaths --value); test -n "$FRAGMENT"; CHECKSUM_FILE=$(mktemp); trap 'rm -f "$CHECKSUM_FILE"' EXIT; sha256sum "$FRAGMENT" $DROPINS > "$CHECKSUM_FILE"; sudo systemctl cat <UNIT>; sudo systemd-analyze verify "$FRAGMENT"; <APPLICATION_CONFIG_VALIDATOR>; sha256sum -c "$CHECKSUM_FILE"; sudo systemctl daemon-reload; sudo systemctl reset-failed <UNIT>; sudo systemctl restart <UNIT>; sudo systemctl is-active <UNIT>

Verify and recover

Verification

Evidence required before you close the incident.

  1. Confirm systemctl is-active <UNIT> returns active and systemctl is-failed <UNIT> does not return failed.

  2. Review journalctl -u <UNIT> --since '<RECOVERY_START>' --no-pager for new errors.

  3. Exercise the service's documented local health check and one representative read-only request.

  4. Confirm the host has rejoined its load balancer or dependency pool only after health is stable.

  5. Monitor restart count, error rate, latency, and saturation for at least 15 minutes.

Commands in these steps

  • systemctl is-active <UNIT>
  • systemctl is-failed <UNIT>
  • journalctl -u <UNIT> --since '<RECOVERY_START>' --no-pager

Rollback

The safe way back.

  1. If the controlled restart fails or the service is unhealthy, stop after the single approved attempt, keep the host out of rotation, restore the captured last approved application and unit configuration through configuration management, run the supported configuration validator, perform one additional owner-approved restart, and escalate if it does not recover.

  2. Do not loop restarts or clear the failed state repeatedly.

Escalate when

Conditions that need another owner.

  • The last approved configuration also fails validation.

  • The service is stateful and restart safety cannot be established.

  • A dependency, filesystem, credential, kernel, or resource-exhaustion failure remains after configuration rollback.

  • The unit fails the one controlled restart or multiple hosts show the same deployment failure.

The KBY Brief

Fixed? Get the next one before it hits you: The KBY Brief

Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.

About this page

Credited to

Dates

Published

Last updated

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

Sources

Based on the vendor sources listed below. Check each step against your environment before acting.

First published 22 August 2026 · Last updated 8 November 2026