Simulation Notice
This is a fictional simulation for diagnostic practice. There is no live incident, no real organisation, and no genuine telemetry. All logs, commit histories and file paths below are illustrative and bounded to a controlled lab.
Scenario
A fictional platform team operates a self-hosted CI/CD system for a monorepo. Over the past two working days, the build stage of the main pipeline has completed with a green status, yet roughly one in five runs leaves the downstream deployment stage unable to locate the expected build artifact. No error is raised at the point of failure; the pipeline simply reports success and the artifact is absent when the next stage attempts to consume it.
The on-call engineer has been asked to determine, using only read-only evidence collection in an isolated lab replica, why a nominally successful build stage is not reliably producing its artifact, and to propose a bounded, reversible next action.
Evidence
Before forming hypotheses, the engineer gathers observable evidence rather than assuming a cause. Three read-only checks are appropriate for this bounded lab:
- Runner service logs for the hour preceding a known failed run, to look for silent warnings that do not fail the job.
- Recent history of the pipeline definition file, to check whether a configuration change coincides with the onset of the problem.
- Available disk space on the runner's workspace volume, since silent artifact loss is a known symptom of a workspace nearing capacity.
Hypotheses and Actions
Two hypotheses are worth testing against the evidence, and neither should be acted on until confirmed:
- Hypothesis A – Workspace capacity exhaustion. If the workspace volume is near full, the build step may write a truncated or zero-byte artifact without the pipeline treating this as a failure, because the write call itself did not error.
- Hypothesis B – Unreviewed pipeline configuration change. If a recent edit reduced a step timeout to below the build's actual duration on a loaded runner, the stage may exit in a state that is reported as success but skips the final artifact copy.
The engineer runs the following read-only commands in the isolated lab replica, each paired with an explicit pass condition before any change is considered:
Reveal
In this scenario, the workspace volume evidence shows usage consistently above ninety per cent capacity immediately before failed runs, while the pipeline definition history shows no unreviewed timeout changes in the relevant window. The confirmed cause is Hypothesis A: intermittent workspace exhaustion, not a configuration regression. The fix suggested for validation is to clear stale build outputs from the workspace and increase the mount allocation, then re-run the pipeline in the lab to confirm the artifact is produced consistently before any change is considered for a real environment.
Learning Outcome
This drill reinforces that a pipeline reporting success is not proof that every step produced valid output; artifact presence must be checked independently. It also demonstrates why evidence collection should precede hypothesis selection, and why a single plausible cause should not be acted on until corroborating evidence rules out the alternative. Before any parallel real change, an engineer should confirm workspace headroom, confirm no unreviewed configuration drift, and re-validate artifact presence for several consecutive runs rather than one. The next safe decision in this fictional case is to schedule a workspace capacity increase and re-test in the lab, keeping the change reversible by retaining the prior mount size as a documented rollback boundary until artifact reliability is confirmed.