Controlling a Bounded Automation Workflow with systemd

Automation is not merely a command that runs without a person. It is an operational agreement about what starts work, which identity performs it, what resources it may touch, how success becomes observable and how an operator can stop or reverse it. This guide uses a deliberately small systemd

The exercise belongs on an isolated, non-production Linux

#1. Learning Objectives

• Explain the distinction between a unit, a service process, a timer activation and evidence that useful work completed.
• Map dependencies, data flow and trust boundaries before changing service-manager state.
• Implement one bounded workflow with a dedicated unprivileged identity and narrowly owned paths.
• Validate definition, activation, execution and output as separate layers.
• Stop on ambiguous evidence, recover the previous state and identify when escalation is safer than experimentation.

#2. Prerequisites

Use a disposable virtual machine or isolated lab host on which systemd is the active service manager. You need shell access and authorised administrative elevation for creating a service account, directories and system units. Administrative access is a capability, not a reason to run the workload as root. Confirm that the host is outside production monitoring, compliance and workload dependencies.

Before changing anything, run systemctl --version, systemctl is-system-running and man systemd.timer. The first records the local implementation, the second reveals whether the manager already reports a degraded state, and the third is the local authority for timer directives. Stop if systemd is absent, the host is unexpectedly production-connected, elevation is unauthorised, the manager is already degraded for unexplained reasons, or the local manuals contradict the proposed unit syntax.

#3. Content

#

3.1 The operating model

A unit is a named object understood by the system manager. A .service unit describes execution; a .timer unit describes when a related service should be activated. Activation means that systemd attempted to start a unit. It does not prove that the program produced the intended output. Enablement establishes a relationship used at boot or another target transition; it is not the same as starting a timer now. A oneshot service performs bounded work and exits, so an inactive state after successful completion may be normal rather than a failure.

The workflow has four evidence layers. First, definition evidence shows whether systemd can parse and load the unit files. Secondly, scheduling evidence shows whether the timer is active and has an expected trigger. Thirdly, execution evidence comes from service status and journal records. Finally, outcome evidence is the state file written by the workload. Accepting only one layer creates blind spots: a listed timer may activate a failing service, while an old output file may survive after scheduling has broken.

#

3.2 Components, dependencies and data flow

The timer asks the system manager to activate the service. The service manager changes credentials to the dedicated account and launches /usr/bin/date. The process writes through a shell to a file in its private state directory, then exits. The service manager records lifecycle evidence in its journal. This chain depends on the unit files, the executable paths, the service account, directory ownership, system time, the system manager and sufficient storage.

The main trust boundary lies between the authorised operator, who may modify system configuration, and the service identity, which should write only its own state. Another boundary lies at the journal: logs may disclose operational details to readers who have journal access. The example uses absolute executable paths to reduce dependence on a mutable search path, but this does not authenticate executable contents. Package integrity remains a host-level assumption.

#

3.3 Designing containment before execution

The service account has no interactive shell and owns only /var/lib/kby-heartbeat. The service unit declares that identity rather than inheriting root. The workload makes a single predictable write. The timer interval is five minutes, limiting frequency while remaining easy to observe. These are design choices, not universal prescriptions; production cadence should derive from workload needs and failure costs.

The following proposed files must be checked against systemd.service, systemd.timer and systemd.exec on the target host. A reviewer should confirm the hardening directives are supported locally.

1[Unit]
2Description=Write a bounded lab heartbeat
3
4[Service]
5Type=oneshot
6User=kby-heartbeat
7Group=kby-heartbeat
8ExecStart=/bin/sh -c '/usr/bin/date --iso-8601=seconds > /var/lib/kby-heartbeat/last-run'
9NoNewPrivileges=yes
10PrivateTmp=yes
11ProtectSystem=strict
12ReadWritePaths=/var/lib/kby-heartbeat

The shell is used only for redirection; it also introduces a parsing boundary. In a production design, prefer a small reviewed executable that opens the destination directly where that reduces ambiguity. Here, fixed literal content and an unprivileged account bound the risk. No network access or credentials are required.

#

3.4 Change sequence and cause-and-effect

Create the dedicated system account and state directory only after confirming the names are unused. Place the reviewed files at /etc/systemd/system/kby-heartbeat.service and /etc/systemd/system/kby-heartbeat.timer, owned by root and not writable by the service account. Run the unit verifier before asking the manager to reload definitions. Verification is a pre-change quality gate; a reload makes definitions visible but does not start the workflow.

After a clean verifier result, reload the manager, then enable and start only kby-heartbeat.timer. This is a state-changing operation: it may create enablement links and immediately schedule activation. The approved scope is the named lab timer. Stop if the command affects any other unit, reports dependency failures, or if manager health worsens. Recovery is to stop and disable the timer, restore or remove the two unit files, reload definitions and confirm no process remains.

#4. Examples

#

4.1 Worked evidence chain

Assume the local verifier returns no diagnostics for both files. That observation supports the narrow inference that the files passed checks implemented by the installed verifier; it does not prove runtime success. After starting the timer, systemctl list-timers kby-heartbeat.timer should show the unit and a next or last trigger appropriate to its state. Exact formatting and timestamps vary, so do not copy a sample timestamp into an incident record.

After an activation, inspect systemctl show kby-heartbeat.service -p Result -p ExecMainStatus. An expected successful observation is Result=success with ExecMainStatus=0. Interpret this as process-level success for the recorded invocation. Then inspect the output with stat /var/lib/kby-heartbeat/last-run and cat /var/lib/kby-heartbeat/last-run. Pass only if the owner and path match the design, the content is a plausible recent timestamp, and its modification time advances after a known activation.

The journal supplies sequencing evidence. A reviewer should correlate the timer activation, service start and completion rather than searching only for the word “error”. Absence of an error string is weak evidence because messages vary. Structured unit filtering, result properties and output inspection form a stronger chain.

#

4.2 Interpreting a failure

Suppose the service result is non-zero and the journal reports permission denied for last-run. The symptom is failed execution; the likely cause is that the service identity cannot write the state path or that a hardening rule blocks it. Diagnose with directory ownership, mode and unit properties before changing policy. Correct the ownership if it violates the design. Do not remove ProtectSystem merely to make the symptom disappear. Retry once, then verify both process and outcome evidence.

#5. Exercises

#

5.1 Objective and setup

Implement the two-unit workflow on the isolated host and demonstrate one scheduled activation. Record the host identifier, installed systemd version, initial manager health, unit-file checksums and the approved scope. Confirm with the lab owner that creating the account and system units is permitted. Take a virtual-machine snapshot if available; otherwise preserve copies of any pre-existing same-named files and stop if such files are unexplained.

1. Confirm that the account, unit names and state path are unused. This prevents accidental takeover of existing resources.
2. Create the system account and its owned state directory using the platform’s authorised account-management procedure.
3. Write the proposed unit files, then compare their contents with the reviewed definitions.
4. Run systemd-analyze verify /etc/systemd/system/kby-heartbeat.service /etc/systemd/system/kby-heartbeat.timer. Proceed only with no unresolved diagnostics.
5. Run systemctl daemon-reload, followed by systemctl enable --now kby-heartbeat.timer. These commands change manager state; limit them to the named units.
6. Observe the timer, service result, journal and output file. Do not repeatedly restart a failing service.

#

5.2 Pass, stop and cleanup conditions

The exercise passes when the timer is active, one invocation records process success, the service runs under kby-heartbeat, and the state file contains a recent timestamp owned by that identity. Preserve commands and actual output as evidence, clearly labelled as observations from the lab.

Stop immediately if an unexpected unit starts, privileged files change, the service runs as root, the output escapes the approved directory, system-manager health degrades, diagnostics are unclear, or the host is discovered to carry production dependencies. Disable and stop the timer before diagnosis. Escalate to the host owner when ownership conflicts, mandatory access-control denials, packaging differences or existing failures cannot be explained safely.

Cleanup is mandatory even after success: stop and disable the timer, remove only the two exercise unit files, reload the manager, confirm the units are absent or not found, remove the lab state directory after preserving required evidence, and remove the dedicated account using the platform’s approved procedure. Never use recursive deletion against a variable or unverified path.

#6. Validation Guidance

Validation should answer separate questions in order. Does the installed implementation recognise the definitions? Is the intended timer enabled and active? Did the service execute under the intended credentials? Did it produce the bounded outcome? Did rollback remove scheduling and execution capability? Record negative evidence as well: no unrelated changed units, no process left running and no writes outside the state directory.

1. Compare systemctl --version and local manual pages with every directive used.
2. Verify unit syntax before reload; unresolved warnings are a stop condition.
3. Inspect systemctl show kby-heartbeat.service -p User -p Group -p FragmentPath to confirm identity and definition source.
4. Inspect timer state and trigger listing without assuming that “active” means successful work.
5. Correlate result properties and unit-filtered journal records for one activation.
6. Inspect output ownership, content and modification time.
7. After cleanup, confirm the timer is disabled and inactive and no matching process remains.

#

6.1 Production bridge

Moving the pattern into production requires a separate change, not promotion by assumption. Replace the demonstration command with a reviewed executable; identify data classification, resource limits, restart behaviour, schedule semantics, monitoring ownership and deployment mechanism. Confirm whether the workload needs network access, secrets or writable paths. Grant only those capabilities, and ensure the service account cannot modify its unit or executable.

Production approval should include peer review, a maintenance window where warranted, tested rollback, alert ownership and an escalation contact. Validate on the actual distribution and installed systemd release. Residual risks include service-manager defects, compromised host binaries, journal exposure, clock anomalies and dependencies outside the unit model. Hardening narrows consequences; it does not establish complete isolation.

#7. Common Mistakes

• Equating enablement with health: enablement expresses future activation relationships. Check runtime and outcome evidence separately.
• Running as root for convenience: this expands consequences. Correct ownership and declare the dedicated identity instead.
• Removing hardening after a denial: diagnose identity, path and policy first; broad relaxation can conceal a design error.
• Trusting stale output: correlate modification time with a known activation and journal sequence.
• Using restart loops as diagnosis: repeated attempts may overwrite evidence or amplify side effects. Stop the timer and inspect one failure.
• Editing vendor unit files: package updates may overwrite them. Use an approved local unit or drop-in strategy and record precedence.

#8. Key Takeaways

• A timer requests activation; it does not prove successful work.
• Definition, scheduling, execution and outcome require distinct evidence.
• A dedicated identity and narrowly writable directory contain foreseeable failure.
• Version and policy checks belong before change because systemd behaviour is locally implemented.
• A safe workflow includes stop conditions and cleanup from the beginning.

#9. Operational Handoff and Recovery Decision

Before closing the lab record, verify that the timer is disabled and inactive, the unit definitions are removed or restored, the manager has reloaded cleanly, no matching process remains and retained evidence contains no sensitive host data. If any check fails, leave the timer stopped, preserve the journal and unit files, avoid broader permission changes, and escalate with the installed version, exact diagnostics, affected paths and rollback actions already attempted.