Prove a Linux Service Runs Correctly Under systemd

Automation and Service Operations work is the discipline of making a defined task run reliably, on its own trigger or schedule, in a way that someone other than the person who wrote it can independently verify. On a Linux

This guide builds that foundation from first principles and then walks through one bounded workflow: defining a small systemd service, starting it under supervision, and gathering the evidence needed to trust the result before it is anywhere near a production host. It assumes you are working in an isolated or non-production environment, that you have confirmed the systemd version and your permission level on that host, and that you can read command output critically rather than assuming success from a lack of error text. Nothing in this guide modifies an existing production unit or requires privilege beyond what a supervised lab account should already hold.

#Learning Objectives

• Explain how systemd units, dependencies and targets represent an automated task in Automation and Service Operations work.
• Identify the trust boundary between a unit file’s declared configuration and the process systemd actually supervises.
• Create, start and stop a bounded systemd service while collecting verifiable evidence at each step.
• Diagnose common systemd failure modes by connecting symptom, cause and correction.
• Carry the same validation reasoning into a production systemd change under the correct permissions and escalation path.

#Prerequisites

• Access to an isolated or non-production Linux host running systemd as its init system (confirm with ps -p 1 -o comm=).
• A confirmed account with permission to manage user-scoped or supervised system units on that host; do not proceed on unverified permissions.
• Basic comfort with the Linux command line and with reading structured log output rather than skimming for the absence of errors.
• Confirmation that no existing production automation on the host already uses the unit name kby-test.service.

#Content

#

The Automation and Service Operations Mental Model

An automation task, in this context, is a piece of work that a system should perform without a human directly typing the command each time. Automation and Service Operations treats that task as something with a defined start condition, a defined body of work, and a defined end state that can be checked. systemd gives this idea a concrete shape called a unit: a small text file describing one thing systemd can manage — a service, a timer, a socket, a mount, or a target that groups other units together.

Three terms matter before anything else. A service unit (.service) describes a process systemd starts, supervises and can restart. A target (.target) is a synchronisation point, similar in spirit to an old-style runlevel, that other units can depend on. A dependency directive such as After=, Wants= or Requires= tells systemd the order and strength of a relationship between units — After= is ordering only, while Requires= also makes the dependent unit’s success a condition for the depending unit. These are facts about systemd’s documented behaviour; the interpretation of what a given dependency means for your specific workflow is something you must reason through for each unit.

#

systemd Components, Dependencies and Trust Boundaries

systemd runs as PID 1, the first userspace process the kernel starts, which gives it the authority to supervise every other process’s lifecycle on the host. When you ask systemd to start a unit, it reads the unit file, resolves its dependencies, forks the described process, places it in a control group (cgroup) for resource and lifecycle tracking, and forwards its standard output and error to the journal. This is the core trust boundary in Automation and Service Operations work: the unit file is a declaration of intent that a sufficiently privileged process (systemd) executes on your behalf, and the privilege required to install that file is separate from the privilege the resulting process runs with.

System-wide units placed under /etc/systemd/system/ require root or equivalent privilege to install and are executed according to the User= and Group= directives inside the file — omit them and the service defaults to running as root, which is rarely necessary for an automation task. User-scoped units placed under ~/.config/systemd/user/ and managed with systemctl --user run inside your own login session and never require elevated privilege to create or manage, which is why this guide uses that path for the bounded exercise: it keeps the entire workflow inside least-privilege boundaries while still exercising the same dependency, supervision and evidence-gathering reasoning you would use on a system unit.

#

Risk Boundaries and Assumptions

The commands in this guide are read-only checks or bounded state changes against a single test unit that you create and remove yourself; none of them touch an existing production service. This guide assumes, and you should confirm before proceeding, that your account either has an active login session or lingering enabled for user-scoped units (checked with loginctl show-user $(whoami)), and that the host you are working on is the isolated environment named in the prerequisites rather than a host carrying live automation. If either assumption does not hold, stop and re-confirm your environment before running any state-changing command below.

#Examples

#

Worked Example: Defining and Starting a Bounded Test Service

The first step is to declare the unit. Create ~/.config/systemd/user/kby-test.service with the following content:

1[Unit]
2Description=KBY bounded validation test service
3
4[Service]
5Type=simple
6ExecStart=/usr/bin/sleep 120
7
8[Install]
9WantedBy=default.target

Each line has a specific purpose you should be able to justify before running it. Type=simple tells systemd the main process is the one named in ExecStart= and that systemd should consider the service started as soon as that process launches. ExecStart=/usr/bin/sleep 120 gives the service a harmless, time-bounded body of work — the process exits cleanly after two minutes, so a mistake here cannot leave an orphaned process running indefinitely. WantedBy=default.target only matters if you later enable the unit; for this bounded exercise you will start it directly and never enable it, so nothing changes at your next login.

After saving the file, run systemctl --user daemon-reload so systemd re-reads the unit directory, then systemctl --user start kby-test.service. Immediately afterwards, systemctl --user status kby-test.service should report Active: active (running) together with a process ID and a start timestamp. That status line is your first piece of evidence: it confirms systemd successfully forked the declared process and is actively supervising it, not merely that the command you typed returned without an error.

The second piece of evidence comes from the journal. Running journalctl --user -u kby-test.service --no-pager -n 20 should show log entries timestamped at the moment you ran start, associated with the same unit name. If the status line is active but the journal shows nothing relevant, that mismatch is itself informative — it usually means you are querying the wrong scope rather than that the service failed silently.

#Exercises

#

Exercise: Validate the Full Lifecycle of kby-test.service

Objective: confirm, with direct evidence, that a bounded systemd service starts, runs and stops exactly as declared, and that no trace of it remains once you finish.

Setup: on your isolated host, confirm systemctl --user status kby-test.service currently reports the unit as not found. This is your clean baseline; do not continue if a unit with this name already exists.

Steps and expected evidence: create the unit file exactly as shown in the worked example, run daemon-reload, then start the unit. Expected evidence is an active, running status with a process ID, plus matching journal entries. Wait for the two-minute sleep to finish naturally and re-check status; expected evidence is now Active: inactive (dead) with a clean exit code of 0.

Pass condition: you observed active-running status with a valid PID, corresponding journal entries, and a clean inactive-dead exit after the sleep duration elapsed, without needing to force-stop the process.

Stop condition: if daemon-reload or start returns an error, or if status shows repeated activating (auto-restart) cycling, stop and move to the diagnosis steps in Common Mistakes before repeating the start command.

Cleanup: once you have gathered your evidence, run systemctl --user stop kby-test.service if it is still active for any reason, delete ~/.config/systemd/user/kby-test.service, and run systemctl --user daemon-reload again. Confirm cleanup with a final systemctl --user status kby-test.service, which should return to reporting the unit as not found.

#Validation Guidance

Validating a systemd task means treating each status line and log entry as evidence to be interpreted, not as a pass or fail flag to glance at. A unit reporting active (running) tells you systemd successfully launched and is tracking the process; it does not by itself tell you the process is doing useful work, so pairing status with journal output — which shows what the process actually said — is the minimum bar for trusting an automation task. The validation steps in this guide are ordered so each one either confirms the previous step’s claim with independent evidence or establishes a clean baseline you can return to.

Before extending this reasoning to a real automation task, decide in advance what “working” means for that specific unit: an expected exit code, a specific log line, a file the process should create, or a downstream effect it should trigger. Without that decision made explicitly beforehand, it becomes easy to interpret an ambiguous status as success simply because nothing obviously failed.

#Common Mistakes

• Symptom: systemctl --user start fails with “Failed to connect to bus”. Cause: no active D-Bus user session, typically a bare SSH session without lingering enabled. Diagnosis: run loginctl show-user $(whoami) and check the Linger value. Correction: enable lingering with loginctl enable-linger or run the exercise from a full interactive login session, then retry.
• Symptom: status repeatedly cycles through activating (auto-restart). Cause: the process named in ExecStart= exits immediately with a non-zero status. Diagnosis: read the exit code and stderr in journalctl output for the unit. Correction: fix the ExecStart= line, run daemon-reload, and start again.
• Symptom: systemctl --user status reports the unit as not found after you edited the file. Cause: daemon-reload was not run after the change, so systemd is still using its cached unit table. Diagnosis: compare the file’s modification time with the last reload. Correction: run daemon-reload and re-check status.
• Symptom: permission denied writing the unit file. Cause: attempting to place a system-wide unit under /etc/systemd/system/ without the required privilege, when a user-scoped unit was sufficient. Diagnosis: confirm which directory the exercise actually requires and your permission on it. Correction: use the per-user path shown in this guide, or request supervised elevated access through the correct change process if a system unit is genuinely required.

#Production Bridge

The reasoning above transfers directly to production, but the permission and safety boundaries tighten. A production-facing automation task is more likely to live as a system unit under /etc/systemd/system/, which means installing or changing it requires an authorised change process, not just technical capability. Before touching a production unit, confirm the exact systemd version on that host, review the unit’s existing User=, ProtectSystem= and NoNewPrivileges= directives so you understand the privilege it already runs with, and confirm a rollback path — typically the previous unit file content plus a recorded daemon-reload and restart — before making any change.

Escalate rather than proceed if any of the following are true: the unit you need to change supervises a service with an existing incident or active dependency you cannot fully map, you cannot confirm the systemd version or distribution defaults in use, or the change would require privilege beyond what your role has been explicitly granted. In each case, the safe action is to document the intended change and the evidence gathered so far, and hand off to whoever holds the required authority, rather than applying an unverified change under time pressure.

#Key Takeaways

• A systemd unit is a declaration of intent; trusting an automation task requires evidence from both status and journal output, not status alone.
• User-scoped units let you exercise the full supervision and dependency model at least privilege, which is why this guide’s bounded exercise never required root.
• Every state-changing command in this workflow has a defined rollback: stop the unit, remove the file, reload, and confirm the unit is gone.
• The same evidence-first validation habit — confirm baseline, apply one change, gather independent evidence, confirm the end state — is what separates a supervised systemd change from a guess that happened to work.

Carrying this habit forward, treat every systemd change you make outside a lab the same way: confirm the version and your permission first, define what evidence would prove success or failure before you act, and keep a recorded rollback path ready so a stop condition never becomes an unplanned outage.