From Timer to Evidence: Operating a Bounded systemd Workflow

Automation is not merely a command that runs without a person present. It is an operational agreement covering what initiates work, which identity performs it, what dependencies must exist, where evidence appears and how an operator contains failure. This guide builds that agreement around a disposable systemd

The example is bounded rather than production-ready. The supplied primary source establishes that the systemd manuals document unit behaviour, service management and operational configuration, but it does not verify a particular local release or distribution policy. Confirm support against the manuals installed on the validation host before acting. Facts, local observations and recommendations are labelled through context: documented concepts come from the cited manual collection, command output is evidence to observe locally, and production controls are recommendations requiring human review.

#1. Learning Objectives

• Explain how a timer unit, service unit, executable and journal form an automation chain.
• Separate activation, execution, outcome evidence and operational health rather than treating “enabled” as “working”.
• Implement a disposable workflow with explicit permissions, stop conditions and a recovery path.
• Interpret unit state, journal records and an output artefact as complementary evidence.
• Transfer the laboratory pattern to production only after ownership, security and failure impact have been reviewed.

#2. Prerequisites

• An isolated or non-production Linux
  The KBY Lexicon

  Linux

  Linux is the open-source kernel underlying most server, container and cloud infrastructure; distinct from the distributions built around it.

  Read Full Context
  host that uses systemd as its service manager.
• A shell account permitted to inspect systemd and, for the implementation stage only, authorised access to create files under /etc/systemd/system, run systemctl daemon-reload and control the disposable units.
• Approval to create /var/lib/kby-heartbeat. Use a different approved path if local policy reserves /var/lib.
• Basic familiarity with shell quoting, file permissions and reading command output.
• A recorded pre-change observation showing that kby-heartbeat.service and kby-heartbeat.timer do not already exist.

The exercise assumes a system-level manager and therefore involves privileged state changes. Least privilege means separating preparation from approval: draft files as an ordinary user, inspect them, and use authorised elevation only for the specific installation and control actions. The residual risk is not zero. A malformed unit can still consume resources, collide with local policy or confuse monitoring, so the payload is intentionally short, local and disposable.

#3. Content

#

3.1 The systemd mental model

A unit is a named object managed by systemd. Different suffixes represent different kinds of object. Here, .service describes a process invocation and .timer describes when the matching service should be activated. A oneshot service performs bounded work and exits instead of remaining as a daemon. A unit state is systemd’s view of that object; it is not, by itself, proof that the intended business outcome occurred.

The timer is the trigger, not the workload. When its activation condition is met, the manager starts the associated service. The service asks /usr/bin/date to append one line to a designated file. systemd records lifecycle messages in the journal, while the file records the intended payload outcome. These evidence channels answer different questions: timer metadata indicates scheduling, service state indicates process handling, journal entries explain lifecycle events, and the heartbeat file demonstrates the requested write.

The trust boundaries matter. The privileged operator controls unit installation and manager state. The manager interprets unit configuration. The configured service identity receives only the filesystem permissions needed for its output directory. The executable path is absolute so shell path lookup does not decide which programme runs. The output file is evidence, but it is not trustworthy against an account allowed to rewrite it; production assurance may require central logs, integrity controls or independent monitoring.

#

3.2 Dependencies and cause-and-effect

Ordering and requirement relationships are distinct. Asking one unit to start after another controls sequence; it does not necessarily express what should happen if the other unit is unavailable. This laboratory workflow has no network dependency and writes only to a local directory. That omission is deliberate: every external dependency expands the failure domain and requires its own timeout, credential, retry and recovery design.

The timer uses a calendar expression for a periodic minute boundary and Persistent=false. The intended laboratory behaviour is therefore to wait for a future activation rather than catch up after downtime. Calendar syntax and directive availability are version-sensitive; verify both with the host’s installed systemd.timer documentation. Random delay is included to demonstrate that scheduled time and exact execution time can differ. Production operators must account for that when defining an alert threshold.

#

3.3 Design before action

Write a compact change record before implementation: the two unit names, output path, service identity, expected activation window, success evidence, maximum observation period and cleanup owner. Capture the installed systemd version and distribution release as environmental observations, not as universal claims. Then inspect whether the names already resolve. A “not found” result is the required baseline for this exercise; any existing unit is a stop condition, not an invitation to overwrite it.

The proposed service account should be a dedicated, non-login system account created through the host’s approved identity mechanism. Account creation varies materially by distribution and is intentionally not encoded as a universal command here. A human reviewer must confirm the local procedure, identifiers, home-directory policy and lifecycle. The directory should be owned by that account and inaccessible to unrelated users. If your laboratory cannot provide this boundary, do not substitute root; choose a user-level systemd exercise or escalate for an approved sandbox.

#4. Examples

#

4.1 Worked unit pair

After the account and directory have been approved and prepared, draft these files outside the unit search path. The service has no shell pipeline: date receives the output file as an argument using its own formatting capability. Confirm locally that the executable accepts the shown option before installation.

1[Unit]
2Description=Write a bounded laboratory heartbeat
3
4[Service]
5Type=oneshot
6User=kby-heartbeat
7Group=kby-heartbeat
8ExecStart=/usr/bin/date --utc +%%Y-%%m-%%dT%%H:%%M:%%SZ
9StandardOutput=append:/var/lib/kby-heartbeat/heartbeat.log
10StandardError=journal
11NoNewPrivileges=yes
12PrivateTmp=yes
13ProtectSystem=strict
14ProtectHome=yes
15ReadWritePaths=/var/lib/kby-heartbeat

The doubled percent signs are intended for systemd’s unit parsing before date receives its format. This is a version-sensitive detail that must be checked with the installed manual and static verifier. The hardening directives attempt to prevent privilege gain, isolate temporary storage, make most of the filesystem read-only, hide home directories and grant a narrow write exception. They reduce exposure but do not prove complete containment.

1[Unit]
2Description=Trigger the bounded laboratory heartbeat
3
4[Timer]
5OnCalendar=*-*-* *:*:00
6RandomizedDelaySec=10s
7Persistent=false
8Unit=kby-heartbeat.service
9
10[Install]
11WantedBy=timers.target

Save the drafts as kby-heartbeat.service and kby-heartbeat.timer. First run the host’s unit-file verification tool against the draft paths. No reported error is the expected evidence, although static checking cannot prove runtime permissions or the payload outcome. Have a reviewer compare the paths, identity and directives with the change record before privileged installation.

#

4.2 Interpreting output

After authorised installation and manager reload, manually start the service once before enabling the timer. This separates payload correctness from schedule correctness. A successful test should append one UTC line, produce no unexpected error in the journal and leave the oneshot service inactive after completion. Inactivity after a successful oneshot run is an observation consistent with bounded completion; it must not be mislabelled as a failed daemon merely because it is no longer running.

Next, start the timer and inspect its timer listing. An active timer with a future activation demonstrates that scheduling was accepted. Wait only for the pre-agreed observation window, then compare the heartbeat timestamp and journal entries. Pass requires both activation evidence and a new output line. If either channel disagrees, stop and diagnose rather than repeatedly restarting the unit.

#5. Exercises

#

5.1 Objective and setup

Your objective is to produce two successful, attributable heartbeat writes: one from a manual service start and one from timer activation. Use a snapshot-capable disposable virtual machine where available. Record the initial state, obtain approval for the dedicated account and directory, and retain copies of the reviewed drafts outside /etc/systemd/system.

1. Observe: record the local systemd version, executable location, unit search assumptions and absence of both names. Purpose: prevent accidental collision. Expected evidence: version output, executable metadata and “not found” unit queries.
2. Verify drafts: run the static unit verifier on both draft files. Purpose: detect parsing and dependency mistakes before manager state changes. Expected evidence: no verifier errors. Stop on any diagnostic you cannot explain.
3. Install through an authorised reviewer: copy only the approved files into the confirmed system unit path with root ownership and non-writable-by-others modes. Purpose: protect configuration integrity. Expected evidence: file metadata matches policy.
4. Reload manager configuration: run systemctl daemon-reload. Purpose: make the manager re-read unit definitions. Expected evidence: subsequent status output identifies the intended files. This changes manager state but does not start the workload.
5. Test the service: start only kby-heartbeat.service. Purpose: validate identity, filesystem access and executable behaviour independently. Expected evidence: one new line, a successful result and relevant journal records.
6. Test automation: start, but do not yet enable, kby-heartbeat.timer. Purpose: make automatic activation temporary and easy to contain. Expected evidence: the timer is active and another line appears within the agreed schedule plus delay and tolerance.
7. Clean up: stop the timer, remove the disposable unit files through the authorised process, reload the manager, and remove the laboratory data and account only after confirming they are exclusively owned by this exercise.

Pass condition: both writes contain plausible UTC timestamps, each corresponds to observed service activity, the timer is visible while under test, and final cleanup leaves neither unit resolvable. Stop condition: stop the timer immediately if writes repeat unexpectedly, resource use grows, output appears outside the approved directory, another workload refers to the units, or permissions differ from the reviewed design.

#6. Validation Guidance

Validation should answer four separate questions. First, did systemd parse and load the intended files? Second, did the service run under the intended identity and terminate successfully? Third, did the timer activate it in the expected window? Fourth, did the payload create the intended result without effects elsewhere? A single green status cannot answer all four.

• Configuration evidence: inspect the loaded unit path and static verification result. Reject shadowed or unexpected unit files.
• Control-plane evidence: inspect timer state and its next or last activation. Interpret random delay before declaring lateness.
• Execution evidence: inspect service result and journal entries for the bounded observation period. Avoid unbounded log queries that disclose unrelated host data.
• Outcome evidence: inspect only the approved heartbeat file, its owner, mode, line count and latest timestamp.
• Negative evidence: confirm no unexpected network dependency, privileged child or write outside the permitted directory was introduced. The basic exercise does not prove every negative; stronger tracing requires separately approved tooling.

Keep evidence free of secrets and unrelated production records. Record command, time, host identifier, expected result, actual result and interpretation. Where evidence conflicts, preserve it and escalate rather than editing the record to fit the expectation. Successful cleanup is also an outcome: both unit names should become unresolved after removal and reload, while no unrelated unit should change state.

#

6.1 Production bridge and residual risk

Moving this pattern to production requires a new review, not merely changing the executable. Identify the business owner, service owner, approving role, maintenance window, alert receiver, data classification and maximum tolerable missed run. Verify directives against the exact deployed version and distribution documentation. Use configuration management

Service credentials, if later required, must not be embedded in unit files because unit configuration and diagnostics may be visible beyond the intended process. Use an organisation-approved secret delivery mechanism and restrict both read access and renewal authority. Consider rate limits, timeouts, idempotency and duplicate execution. A timer may activate again after an ambiguous failure, so a production payload must make retries safe or detect completed work.

Escalate when recovery would affect another service, when the unit participates in boot-critical targets, when an identity or filesystem boundary cannot be verified, or when journal evidence suggests manager-wide failure. Do not disable security controls, delete system-wide state or reboot merely to clear the symptom. The production rollback owner should have a known-good prior configuration, evidence that dependent units tolerate reversal and authority to invoke the change process.

#7. Common Mistakes

• Symptom: the timer is active but no useful output exists. Likely cause: schedule evidence was mistaken for payload evidence. Correction: inspect service result, journal and directory permissions separately.
• Symptom: the oneshot service appears inactive after execution. Likely cause: bounded completion is being judged like a long-running daemon. Correction: inspect the recorded result and output timestamp.
• Symptom: edited configuration appears ignored. Likely cause: the manager was not reloaded or another file shadows the draft. Correction: inspect the loaded path, reload after review and revalidate.
• Symptom: a write succeeds only as root. Likely cause: the directory boundary is wrong. Correction: repair ownership through the approved process; do not run the workload as root.
• Symptom: an expected run seems late. Likely cause: random delay, calendar interpretation or clock state was omitted from the expectation. Correction: compare timer metadata with the agreed tolerance and local time configuration.
• Symptom: cleanup would remove unfamiliar files or an account used elsewhere. Likely cause: the exercise was not isolated. Correction: stop, preserve evidence and escalate; never guess ownership.

#8. Key Takeaways

• A systemd timer defines activation; the associated service defines execution; neither alone proves the intended operational result.
• Validate parsing, loading, execution, scheduling and payload evidence as separate layers.
• Use a dedicated identity, absolute executable paths and a narrow writable directory instead of broad privilege.
• Test the service manually before testing the timer so failures remain attributable.
• Contain changes with unique names, a short observation window, explicit stop conditions and verified cleanup.
• Treat installed manuals, local policy and observed host state as the authority for version-sensitive decisions.

Before any production adaptation, confirm the exact unit files, delegated identity, schedule tolerance, monitoring route and known-good rollback with their owners. Proceed only when a manual run, a scheduled run and a cleanup rehearsal all produce attributable evidence without widening permissions or affecting another service.