Replacing Manual 605 Work with a Verifiable Linux Workflow

#Current Method

Category 605 is the internal work-item classification used in this assignment for a recurring operational task that currently depends on a human operator logging into one or more Linux

Declared assumption (requires confirmation): the exact business definition, SLA and ticketing scope of Category 605 was not supplied with this assignment. This playbook treats it as a generic, repeating, script-driven administrative task on Linux. Before adopting this workflow as an operational standard, the team responsible for the 605 queue must confirm that its scope matches the assumptions here.

The manual method creates three recurring problems rather than one-off incidents. First, there is no consistent, timestamped evidence that a given run happened, succeeded, or was checked before the ticket was closed; closure often depends on the operator’s word rather than a log artefact. Second, execution order and command flags drift between operators and between shifts, because the procedure lives in a wiki page or in someone’s memory rather than in a version-controlled definition. Third, detection of failure is reactive: a downstream system or a customer notices the missing effect of the 605 work days later, at which point the audit trail is thin. None of this is a single break-fix event; it is a structural gap between “a human ran something” and “the system can prove what ran, when, and with what result.”

#Improved Workflow

The redesign keeps the underlying 605 task logic unchanged and replaces the trust boundary around its execution. Instead of an operator running commands interactively with their own broad login privileges, a dedicated, low-privilege system account executes the task through a systemd

#Guardrails

• Least privilege by default: the service account has no interactive shell, no sudo entry, and filesystem access limited to the paths the task genuinely needs, bounding the impact of a compromised or misbehaving script.
• Version-controlled unit files: both unit files live in the same repository as the underlying 605 script, so a change to scheduling or execution logic is reviewed the same way as a code change, not applied ad hoc on the host.
• Environment segregation: identical unit definitions are validated on a non-production host before any production install, so the promotion decision is based on observed behaviour, not on the definitions alone.
• Jitter and persistence: RandomizedDelaySec avoids synchronised load if multiple 605 workers exist, and Persistent=true ensures a missed run during host downtime is executed on next boot rather than silently dropped.

#Validation

Validation happens before, during and after each promotion, not only once at the end.

• Definition parity: diff the installed unit file against the version-controlled source to confirm no drift occurred during manual installation.
• Schedule confirmation: confirm the timer is enabled and shows a correct next-run time rather than assuming activation succeeded.
• Execution evidence: confirm the first automatic run recorded an exit code of zero and the expected output in the journal, not merely that the timer fired.
• Idempotency check: confirm that triggering the service unit a second time in the non-production environment does not duplicate side effects on the 605 work item, since a timer misfire or manual re-trigger should be safe.

1# Non-destructive check that the last run produced the expected evidence file
2test -f /var/log/605-worker/last-run-evidence.json && echo "evidence present" || echo "evidence missing: investigate before promotion"

#Common Mistakes

• Enabling the timer directly in production without a non-production soak period. This removes the only opportunity to observe failure behaviour before it affects live 605 tickets.
• Granting the service account broad or temporary elevated permissions to unblock a stuck run, then forgetting to revoke them. This quietly widens the trust boundary the whole design was meant to narrow.
• Omitting Persistent=true, so a host reboot or maintenance window during the scheduled slot causes a silently skipped run with no alert and no evidence gap flagged.
• Treating a single successful run as proof of reliability rather than observing several cycles, including one that spans a host restart, before trusting the automation with unattended production closure of 605 tickets.

#Recovery

Scenario 1 — service exits non-zero. Symptom: the timer fires but the journal shows a failed unit. Likely cause: a script dependency or input assumption that held in non-production did not hold on this host. Diagnostic evidence: journalctl -u 605-worker.service showing the failing step and exit code. Bounded correction: fix the script or its input handling in version control and re-test in non-production. Rollback: disable and stop the timer, remove the installed unit files, and reload the manager so the host returns to its pre-automation state; resume manual handling of pending 605 tickets. Post-recovery verification: confirm systemctl status 605-worker.timer shows inactive and that no stray unit file remains under /etc/systemd/system/.

Scenario 2 — stale definition after edit. Symptom: a unit file was edited but systemd still behaves according to the old definition. Likely cause: systemctl daemon-reload was skipped after the edit. Diagnostic evidence: systemctl cat 605-worker.service differs from the file on disk. Bounded correction: run systemctl daemon-reload and re-verify. Rollback: not required if the stale definition caused no harmful action; if it did run an unintended step, follow Scenario 1’s rollback.

Scenario 3 — permission denied. Symptom: the journal shows a permission-denied error from the service account. Likely cause: the account’s scoped access does not cover a path the script actually needs. Diagnostic evidence: the specific denied path in the journal entry. Bounded correction: grant the minimum additional access to that specific path only, re-test in non-production, and update the hardening directives in version control. Rollback: disable the timer and revert to manual handling until the minimal permission set is confirmed correct, rather than granting broad access under time pressure.

#Measurable Outcome

Baseline: before change, measure the existing manual handling time and rework rate for Category 605 tickets using the ticketing system’s own timestamps (time opened to time closed, and count of tickets reopened due to a missed or incorrect manual step) over a defined recent period. No specific baseline figures are asserted here; they must come from the organisation’s own ticket history.

Success signal: an automated run completes with exit code zero and its evidence artefact is attached to the ticket without manual intervention, across a defined observation window (for example, a run of consecutive scheduled cycles agreed with the approver, including at least one cycle spanning a host restart).

Measurement method: a journal query counting successful versus failed automated runs, cross-referenced against ticket-system timestamps for evidence attachment and closure time.

Review cadence: weekly during the first four weeks of non-production and early production use, then monthly once the decision threshold below is met.

Decision threshold: full production reliance is only justified once the agreed number of consecutive clean cycles is reached with zero unresolved guardrail violations (permission escalations, missed-run gaps, or evidence-attachment failures); otherwise, revert to manual handling and log the specific findings for the next attempt.

#Adoption and Rollout Checklist for Category 605 Linux Automation

• Business definition and scope of Category 605 confirmed with the responsible team (outstanding until confirmed).
• Non-production host identified and systemd version confirmed.
• Dedicated non-interactive service account created with itemised, minimal permissions.
• Service and timer units version-controlled, peer-reviewed, and installed on non-production only.
• At least one full automatic run observed with exit code zero and evidence artefact present.
• Rollback path tested: timer disabled and unit files removed cleanly with the host returned to its pre-automation state.
• Approver has reviewed the agreed number of clean cycles, including one spanning a restart, before production promotion.
• Weekly review scheduled for the first four weeks of live use.