Designing a Bounded systemd Service with Evidence and Recovery

Automation becomes operationally useful when an operator can explain what initiates it, which identity performs it, what state it may change, what evidence it emits and how that change can be reversed. A systemd

This guide develops one bounded workflow: a system service writes a fixed marker into a dedicated laboratory directory. The marker has no production value; its purpose is to make cause and effect observable without touching an application or network service. The exercise assumes a disposable or isolated Linux

#1. Learning Objectives

After completing the guide, you should be able to:

• distinguish a unit definition, the system manager, a service process, dependency ordering and persistent output;
• describe the data flow and trust boundaries from an operator request to a changed file;
• design an idempotent, narrowly scoped oneshot workflow with observable success;
• separate configuration validity, manager state, process completion and useful outcome evidence;
• diagnose common failures without immediately repeating a state-changing action; and
• remove the laboratory workflow and identify the approvals required for production adoption.

#2. Prerequisites

Use an isolated virtual machine, training host or equivalent non-production environment. Confirm that PID 1 is systemd and consult the manual pages installed on that host. The supplied primary source establishes that the systemd manuals document unit behaviour, service management and operational configuration, but it does not verify the exact release or distribution policy on your machine.

You need shell access, permission to inspect system units and explicitly delegated authority to create one unit file under the system unit search path used by the laboratory host. The worked example assumes /etc/systemd/system, which must be checked locally before use. Root authority is deliberately not presented as an unexplained prerequisite: if your role lacks the narrow permission, stop and ask the environment owner to perform or approve the change. Do not broaden your privileges merely to finish the exercise.

Record a baseline before acting: the installed systemd version, whether a unit named graduate-marker.service already exists, and whether /var/tmp/graduate-systemd-lab/marker.txt exists. If either named object belongs to another exercise or owner, stop and choose an approved unique name. This ownership check prevents rollback from removing unrelated state.

#3. Content

#

3.1 A first-principles model

A unit is a resource description understood by systemd. A service unit describes process execution and lifecycle expectations. The manager loads unit definitions, constructs a transaction and supervises execution. A oneshot service represents a task expected to finish rather than remain as a long-running daemon. Whether the exact directives below behave as described must be checked against the installed systemd.service and systemd.unit manuals.

Three relationships are easy to confuse. Requirement dependencies express whether another unit is pulled into a transaction. Ordering dependencies express sequence, not success propagation by themselves. Installation metadata influences what links are created when a unit is enabled; it is not an unconditional runtime trigger. This exercise starts the unit explicitly and does not enable it, avoiding an unneeded persistence mechanism.

The workflow crosses several trust boundaries. A human submits a request through systemctl. The system manager, which operates with greater authority than an ordinary account, reads administrator-controlled configuration. It then launches a process under the identity declared by the unit. That process attempts to change the filesystem. Finally, the operator interprets manager status, journal records and the resulting file. A clean exit is evidence about the process, while correct file content is evidence about the intended outcome; neither should be substituted for the other.

The first learning image should appear here to reinforce the separation between control input, privileged orchestration, workload identity and evidence. It is explanatory rather than proof of any specific deployment.

#

3.2 Designing the bounded change

The task writes only one known string to one dedicated location. Repeating it produces the same content, so the useful outcome is idempotent even though timestamps and journal entries may change. A dedicated service account limits the process identity, while systemd sandboxing directives can reduce filesystem access. Those controls are defence in depth, not proof of complete isolation: kernel, distribution and systemd-version support require local review.

The proposed unit is:

1[Unit]
2Description=Graduate laboratory marker writer
3
4[Service]
5Type=oneshot
6User=graduate-marker
7Group=graduate-marker
8ExecStart=/usr/bin/install -D -m 0640 /etc/graduate-marker/source.txt /var/tmp/graduate-systemd-lab/marker.txt
9ProtectSystem=strict
10ReadWritePaths=/var/tmp/graduate-systemd-lab
11PrivateTmp=yes
12NoNewPrivileges=yes

Each choice narrows an assumption. An absolute executable path avoids reliance on an interactive shell search path. install creates parent directories and writes with an explicit mode. The source file makes input reviewable rather than embedding shell syntax. User and Group avoid running the payload as root. ProtectSystem and ReadWritePaths express a restricted writable area. Do not copy these directives into production until local manuals and policy tooling confirm support and interaction with mandatory access controls.

#

3.3 Lifecycle and causality

Writing a unit file changes disk state, but an already running manager may retain its previous view. A manager reload asks systemd to rescan unit configuration; it does not itself execute the service. Starting the unit creates a transaction and may run the process. Because the service is oneshot and does not request retained active state, a successful task may subsequently appear inactive. Therefore, “inactive” alone is not a failure: inspect the result, exit status, journal and marker.

Failure containment depends on stopping at the first disagreement. If verification reports a syntax or directive problem, do not reload or start. If the loaded fragment path is unexpected, do not run it. If the start fails, inspect evidence before retrying. Repetition can overwrite clues and is unsafe when a future task is not idempotent.

#4. Examples

#

4.1 Worked interpretation

Assume an authorised administrator has created the dedicated account, source file and unit from an approved change record. The source contains graduate-systemd-lab-v1. A verifier reports no errors; after reload, the manager reports the expected fragment path and declared user. The operator starts the unit once. The result property reports success, the process exit status is zero, and the marker is a regular file containing exactly the approved string with mode 0640 and the expected ownership.

The interpretation is layered. The verifier supports the claim that it found no configuration error in the supplied file; silence should not be inflated into a universal correctness guarantee. The fragment-path observation supports that the intended definition was loaded. A successful result supports normal task completion. Exact content and metadata support the actual outcome. Together they satisfy this laboratory objective. They do not show that a timer, boot target or external dependency would behave correctly, because none is in scope.

A contrasting example is a successful process with the wrong marker content. That is an outcome failure even if systemd records success, because the executable performed its defined operation on incorrect input. Correct the source through the approved change path, rerun verification and execute once more; do not alter the success criterion to match the accidental output.

The second learning image should appear after this example, illustrating an operator comparing configuration, manager and filesystem evidence. It must not imply that a stock photograph is actual test evidence.

#5. Exercises

#

5.1 Objective and setup

Your objective is to implement the marker service on the isolated host and produce an evidence bundle that another practitioner can interpret. Before setup, obtain approval for the unit name, account, source path and output path. Save the baseline observations. Create the unprivileged graduate-marker account without interactive login using the host’s approved account-management process; exact account commands are omitted because they are distribution-sensitive and security-relevant.

Create /etc/graduate-marker/source.txt with the exact approved marker and restrictive administrator-controlled ownership. Create the unit file shown above only after checking executable paths and directive support locally. These are state-changing actions. Their scope is the dedicated account, source, unit and output paths; their risk is unintended privileged execution or filesystem modification if names, ownership or directives are wrong.

#

5.2 Execute with gates

1. Run the unit verifier against the file. Expected evidence is no reported error concerning this unit. If an error, unknown directive or unsafe path appears, stop before reload and consult the installed manual.
2. Request a manager reload through the approved privileged interface. Then inspect the unit fragment path and declared execution identity. Stop if either differs from the reviewed definition.
3. Start the unit once. Do not enable it. Capture result, exit status and recent unit-specific journal entries without including unrelated sensitive logs.
4. Inspect the marker as an ordinary read-only operation. Confirm exact content, regular-file type, expected owner and group, and mode 0640.
5. Record pass only when all layers agree. A warning, unexpected write, ownership mismatch or missing evidence is a stop condition rather than a partial pass.

#

5.3 Pass, stop and cleanup conditions

Pass: the manager loaded the reviewed file, ran under the dedicated identity, reported successful completion and produced only the approved marker with exact content and metadata. Stop: any pre-existing name collision, unsupported directive, unexpected dependency, wrong fragment path, privilege escalation, write outside the laboratory directory or inability to explain the evidence. Preserve diagnostics and escalate to the host owner rather than improvising broader access.

Cleanup: after evidence capture, remove the output, unit and source only if the baseline proved they were created solely for this exercise. Reload the manager and confirm that the unit is no longer found. Remove the dedicated account only when no file, process or approved exercise depends on it. If ownership is uncertain, leave the artefact in place, label it and escalate; uncertain deletion is not recovery.

#6. Validation Guidance

Validation should answer both “did systemd accept and execute the definition?” and “did the intended bounded result occur?”. Capture commands, timestamps, return codes and concise observations in a change record. Redact host identifiers or journal content where organisational policy requires it. Evidence is strongest when it is reproducible, attributable and collected immediately after the controlled action.

Use read-only inspection before and after every state transition. Compare the loaded fragment with the approved file, inspect dependency information for unexpected relationships and check filesystem state independently of manager status. For a production bridge, add peer review, configuration management

Production readiness also requires a meaningful service-level outcome. A marker file is sufficient only for this lesson. A real backup, synchronisation or data-processing task needs domain-specific integrity checks and a response to partial completion. Escalate when the task touches shared data, credentials, network endpoints, regulated records or boot-critical dependencies, or when rollback could remove valid output.

#7. Common Mistakes

#

7.1 Treating ordering as health

Placing one unit after another describes sequence under applicable transactions; it does not automatically prove that the earlier service produced a valid business outcome. Add an explicit dependency only when the operational relationship requires it, then define separate outcome evidence.

#

7.2 Enabling a task that only needs a controlled run

Enablement creates persistence links according to installation metadata. It is unnecessary here and broadens the change. Explicit start is easier to contain and recover. If scheduling is later required, review a timer as a separate workflow with its own trigger, missed-run and concurrency semantics.

#

7.3 Using root to bypass ownership design

A task working as root may conceal missing permissions and increases impact. Begin with the minimum service identity and grant only the required path access. If a privileged preparation step is unavoidable, separate it, document its boundary and obtain approval.

#

7.4 Assuming status output proves the result

Manager state is one observation. A oneshot unit may be inactive after success, and a process can exit zero after writing incorrect input. Check result properties, journal context and domain output independently.

#

7.5 Retrying before diagnosis

An immediate retry can overwrite output or make event ordering harder to interpret. Freeze further change, collect status and journal evidence, compare the loaded unit and inspect permissions. Retry once only after identifying and correcting a bounded cause.

#8. Key Takeaways

• A systemd workflow joins declarative configuration, a privileged manager, a process identity, dependencies and external state.
• Parse evidence, loaded-state evidence, execution evidence and outcome evidence answer different questions.
• Least privilege and explicit writable paths reduce impact but leave residual platform and policy risk.
• A safe exercise has a unique scope, approval, observable pass criteria, stop conditions and ownership-aware cleanup.
• Production transfer requires version confirmation, peer review, delegated permissions, monitoring and domain-specific integrity checks.

Before making the next operational decision, verify the installed manuals, retain the baseline and evidence bundle, confirm that cleanup affects only laboratory-owned objects, and escalate any unexplained manager state, filesystem change or security-policy denial rather than widening permissions or repeating execution.