Replacing Manual Code-First Infrastructure-as-Code Work with a Verifiable Infrastructure as Code Workflow

#Current Method: Manual Code-First Infrastructure as Code Work

Many platform teams already treat infrastructure as code in name, but the daily practice remains largely manual. A practitioner edits a Terraform configuration locally, runs terraform plan against the shared remote state, and applies the change directly from a workstation once the diff looks acceptable. Observation: this pattern accelerates early-stage work because there is no pipeline to configure. Inference: the same pattern removes the natural checkpoints that would otherwise catch destructive diffs, credential misuse, or state corruption before they reach shared infrastructure.

The operating context includes dependencies that are easy to overlook. The remote state backend is a shared, stateful resource; two operators applying concurrently without locking can corrupt it. Provider plugins are pinned or unpinned depending on team discipline, so identical configuration can behave differently on different machines. The actor set typically includes the engineer authoring the change and anyone holding apply credentials to the backend. The trust boundary of consequence sits at the apply credential: whoever can run terraform apply against production state can reshape production infrastructure, regardless of whether the change was reviewed.

Manual code-first work creates recurring friction along three lines. First, evidence is weak: a plan run on one workstation is not preserved, so a reviewer cannot confirm what was actually applied. Second, recovery is ad hoc: without a preserved plan file, a bad apply is diagnosed after the fact by comparing live infrastructure to configuration. Third, permissions tend to accumulate, because manual apply is convenient and teams grant broad apply access to more people than the change volume justifies, widening the blast radius of any single mistake.

Assumption made visible: this playbook assumes Terraform is the implementation platform and that a version control system and at least one non-production environment are available. Where those are absent, the guardrails and implementation steps below require adaptation before adoption.

#Improved Workflow: A Verifiable Terraform Pipeline

The improved workflow keeps the same authoring step but inserts deliberate checkpoints: a machine-checkable validation stage, a preserved and reviewable plan artefact, and an apply stage bounded to an isolated environment before it is ever pointed at shared or production state. Each stage consumes a specific input, changes a specific piece of state or control, and produces evidence the next stage depends on.

Writing accepts a change request and produces a versioned configuration diff. Validation accepts that diff and checks syntax and internal consistency without contacting any provider API; it changes nothing. Planning accepts the validated configuration and current remote state, changes nothing in the infrastructure itself, and produces a persisted plan file that every later stage refers to. Review accepts the plan file, not the raw configuration, so reviewers assess the actual proposed change to real state. Apply accepts the reviewed plan file and applies it to an isolated environment first, changing infrastructure only there. Verification accepts the applied state and output and confirms it matches the plan before any promotion.

The trade-off being accepted is speed for evidence: this workflow is slower per change than an uncontrolled manual apply, in exchange for a reviewable, replayable record of what changed, why, and under whose approval.

#Implementation

Prerequisites: an isolated or non-production Terraform workspace; a pinned Terraform version confirmed against the configuration’s required_version constraint; apply credentials scoped to that isolated environment only, distinct from any production credential; and a version-controlled repository holding the configuration.

1. Confirm the Terraform version and initialise providers with terraform init -backend=false. Expected evidence: a successful initialisation message listing resolved provider versions. Stop condition: if provider resolution fails or resolves an unexpected major version, halt and confirm the required_version constraint with the platform owner.
2. Validate configuration syntax and internal references with terraform validate. Expected evidence: a success message with no errors. Stop condition: any validation error halts progress until resolved in configuration, not by disabling validation.
3. Generate and preserve a plan artefact against the isolated environment’s state with terraform plan -out=tfplan. Expected evidence: a plan summary with add, change and destroy counts and a written plan file. Stop condition: any unexpected destroy or replace action against a resource not intentionally targeted halts the workflow.
4. Inspect the preserved plan before requesting review with terraform show tfplan. Expected evidence: a full, human-readable listing of every attribute change. This step is read-only and exists purely to produce reviewable evidence.
5. Apply the reviewed plan to the isolated environment only with terraform apply tfplan. Expected evidence: an apply summary matching the counts recorded in the preserved plan. Stop condition: if the apply summary differs from the preserved plan, stop immediately; this indicates drift since the plan was generated, and the plan must be regenerated, not forced through. Before this step, capture a versioned state snapshot so a restore point exists.
6. Confirm the resulting state matches expectations with terraform state list. Expected evidence: a resource inventory consistent with the applied plan. Stop condition: any resource present in state but absent from configuration, or vice versa, is treated as drift requiring investigation before promotion.

Only after these stages produce clean evidence in the isolated environment should the same reviewed-plan pattern, not the same plan file, be repeated against a shared or production scope, under the same review and evidence discipline.

#Guardrails

• Apply credentials for the isolated validation environment must be distinct from, and narrower than, any production apply credential; a credential valid in both places defeats the isolation.
• Enable state locking on the remote backend so concurrent plan or apply operations cannot corrupt shared state.
• Require the plan file, not the raw diff, as the reviewable artefact; a review of source code alone cannot show what a provider will actually do.
• Treat any plan showing an unexpected destroy or replace action as a stop condition, not a warning to note and continue past.
• Keep the isolated environment’s state backend separate from production’s, so a mistake in validation cannot touch shared infrastructure.

#Validation

• Run terraform validate before every plan; pass condition is a success message with no errors.
• Run terraform show tfplan and compare it against the intended change description; pass condition is that every listed action was anticipated and none are unexplained destroys.
• Re-run terraform plan immediately before terraform apply tfplan; pass condition is that Terraform confirms the saved plan is still current.
• After apply, run terraform state list and review outputs; pass condition is that resource counts and outputs match the plan preserved before apply.

#Common Mistakes

• Applying directly from a freshly generated plan without inspecting terraform show output first, which allows an unreviewed destroy action through.
• Reusing the same apply credential for both the isolated validation environment and production, which removes the isolation guardrail in practice even though it exists on paper.
• Letting a plan file age past the point where the remote state could have changed, then applying it anyway instead of regenerating the plan.
• Treating terraform init as a one-off setup step rather than re-running it whenever provider or module versions change, which can mask version drift.

#Recovery

Failure mode: apply reports drift from the preserved plan. Symptom: apply counts differ from the plan generated earlier. Likely cause: another operator or automated process changed the target environment after the plan was generated. Diagnostic evidence: compare the pre-apply terraform show tfplan output against a fresh terraform plan run immediately after the failed apply. Bounded correction: cancel the apply if it has not completed; do not re-run the stale plan. Rollback: restore the isolated environment’s state from the backend’s versioned state history to the snapshot taken before the apply, then regenerate a fresh plan from current state. Post-recovery verification: a fresh terraform plan against the restored state shows the expected baseline diff.

Failure mode: provider version mismatch after init. Symptom: terraform init resolves an unexpected provider version, or terraform validate reports new schema errors. Likely cause: an unpinned provider version constraint allowed a newer provider release to be resolved. Diagnostic evidence: terraform providers output compared against the configuration’s required_providers block. Bounded correction: pin the provider version explicitly and re-run terraform init. Rollback: no infrastructure rollback is needed, since this failure occurs before any apply; revert the configuration change if the pin itself was the cause. Post-recovery verification: terraform init and terraform validate both succeed with the pinned version.

For any failure that cannot be resolved with the corrections above, for example suspected state corruption beyond what versioned state history can restore, stop immediately and escalate to the platform or infrastructure owner rather than attempting further Terraform commands against the affected state.

#Measurable Outcome

Baseline: record, for the current manual method, how many applies in the last review period had no preserved plan artefact and how many were applied directly from a workstation without a distinct isolated-environment step. Success signal: every apply against shared or production scope is preceded by a preserved, reviewed plan file and a successful isolated-environment apply using the same configuration. Measurement method: track plan-file presence and review sign-off per change in the version control system or change log; this requires no new tooling beyond retaining plan artefacts and review records. Review cadence: review these counts monthly for the first quarter of adoption, then quarterly once the workflow is stable. Decision threshold: if more than one in ten changes still bypasses the preserved-plan checkpoint after one quarter, treat the workflow as not yet adopted and re-examine guardrail enforcement rather than adding further process.

#Checklist: Adopting the Verifiable Infrastructure as Code Workflow

• Confirm the isolated validation environment has its own state backend and its own apply credential.
• Confirm terraform init, validate and plan are run in that order before every review request.
• Confirm the reviewed artefact is the plan file output, not the source diff alone.
• Confirm apply credentials for production remain separate from the isolated environment’s credentials.
• Confirm a versioned state snapshot exists before every production-facing apply.
• Confirm the next safe decision: do not promote a change to shared or production scope until the isolated-environment apply and verification steps above have produced clean evidence.