Terraform Plan Shows Changes After Console Edits: Fixing Drift Safely

#Current method: why console edits surprise the next plan

TerraformThe KBY LexiconTerraformTerraform is HashiCorp's declarative infrastructure-as-code tool that plans and applies infrastructure changes through provider plugins and a tracked state file. Read Full Context is a declarative tool: you describe infrastructure in configuration, and it provisions to match. Its state records what it last knew. When someone edits a resource in a cloud console, the real object, the state and the code stop agreeing. The next terraform plan compares code with refreshed reality and proposes changes you did not write.

The usual response is to read the diff quickly and either apply it or edit code until the diff disappears. Both are risky. Applying silently reverts an edit that may have been an urgent fix; editing code to hide the diff can adopt a change nobody reviewed. Neither leaves evidence of who changed what or why.

The actors are the engineer running Terraform, the person who made the console edit, and whoever owns the service. The trust boundary is the credential Terraform uses: it can usually modify everything it manages, so an unreviewed apply has a wide blast radius.

#Improved workflow: separate detection from decision

The improvement is to split three questions that a plain plan blends together: what changed outside Terraform, should that change be kept, and how do we make code and reality agree again.

1. Detect with a refresh-only plan. It reports differences between state and real infrastructure without proposing to alter infrastructure. Input: current state and read access. Output: a list of drifted attributes. Trade-off: it needs provider read permissions and may take time on large estates.
2. Classify each drifted resource as keep or revert. This step is a human decision with the change owner, because only they know whether the edit was intended.
3. Reconcile. For keep, update the configuration to match, then confirm a normal plan is clean. For revert, leave the code unchanged and apply a reviewed plan that restores the declared value.
4. Repeat on a schedule so drift is found in days rather than discovered during an unrelated release.

#Implementation

Prerequisites: a non-production workspace or copy of the configuration, read access to the provider, access to the state backend, and the Terraform version in use. Confirm version-specific behaviour, including how refresh-only mode and state refresh are described, in the HashiCorp documentation listed in the sources.

1. Back up state. Pull a copy of the current state to a protected location. Expected evidence: a non-empty file. Stop if the backend is locked by someone else; see safe recovery from a Terraform state lock error rather than forcing the lock.
2. Run a normal plan and save the output as the baseline. This records the surprising diff before anything changes.
3. Run a refresh-only plan. Review each attribute that differs. Expected evidence: drift listed against the specific resources that were edited, and no proposed infrastructure changes.
4. Record a decision per resource with owner, reason and ticket reference.
5. Apply the refresh-only plan only if you want state updated to match reality. This changes state, not infrastructure, which is why you backed it up first.
6. For keep decisions, edit the configuration to match, then run a normal plan. Pass condition: no changes for that resource.
7. For revert decisions, run a normal plan, check it changes only the intended attributes, and have a second person review before applying.

Stop conditions: the plan proposes replacement or deletion of a resource you did not expect, the diff touches resources outside the edited scope, or nobody can identify the owner of an edit. Escalate to a human owner instead of continuing.

#Guardrails

• Use credentials with least privilege; give the scheduled drift job read-only access where the provider allows it.
• Never run an apply from an unreviewed or stale plan file.
• Do not hide drift with ignore rules unless the decision is recorded and reviewed, because they permanently mask future changes to that attribute.
• Keep state backupsThe KBY LexiconBackupA causally disconnected, point-in-time copy of system state, tagged with a consistency marker, that lets you recover from logical corruption or data loss independent of the source system's health. Read Full Context protected, as state can contain sensitive values.
• Rehearse in an isolated environment before touching production.

Assumption for human review: exact command flags, output wording and refresh behaviour vary by Terraform version and provider, so check them against your installed version.

#Validation

1. A normal plan after reconciliation reports no changes for the affected resources.
2. The real resource matches the intended value when inspected in the console or provider tooling.
3. The decision record exists for every drifted resource.
4. A second refresh-only plan shows no drift.

#Common mistakes

• Symptom: the plan reverts an emergency fix. Cause: applying without classifying. Response: classify first and keep the fix in code if intended.
• Symptom: drift reappears the next day. Cause: the editor was not told, or automation also edits the resource. Response: find the source through the provider audit log and restrict console write access.
• Symptom: the diff vanishes after adding ignore rules. Cause: the change was masked, not resolved. Response: remove the rule unless a reviewer approved it.
• Symptom: the plan shows replacement. Cause: a changed attribute forces a new resource. Response: stop and escalate before applying.

#Recovery

If a state update or apply goes wrong, stop further runs. Restore the backed-up state copy using your backend’s documented procedure, then run a plan to see the actual difference. If an apply reverted a setting that should have stayed, restore the value in code and apply a reviewed plan, or reapply it through the console and then reconcile. Verify recovery with a clean normal plan and a refresh-only plan that shows no drift. If the state backend is unavailable or locked unexpectedly, escalate to the backend owner.

#Scheduled drift detection and measurable outcome

Run the refresh-only plan on a schedule, such as daily or weekly, from a pipeline that has read-only access and posts the result to a team channel. Choose the cadence to match how often console edits occur.

Baseline first: count drifted resources over a few weeks before changing anything. Success signals are fewer unexplained plan diffs, shorter time from edit to detection, and every drift item having a recorded decision. Review monthly; if the same resources drift repeatedly, tighten console permissions rather than increasing check frequency. No improvement figures are claimed here, since they depend on your environment.

#Checklist for adopting the drift workflow

1. State is backed up and the backend is not locked.
2. Terraform version and provider permissions are confirmed.
3. Baseline plan is saved.
4. Refresh-only plan reviewed per resource.
5. Keep or revert decision recorded with an owner.
6. Normal plan is clean after reconciliation.
7. Scheduled check is running with read-only credentials.
8. Rollback route is known: restore state copy, then re-plan.

#Prerequisites and permissions in detail

#

Access to request before the first run

Agree the access list with the platform owner before anyone touches state. A drift job needs three separate grants, and they should not be combined into one broad role.

• Provider read access: list and describe permissions for every resource type in the configuration. A missing read permission usually surfaces as a refresh failure for one resource type, not as a clean plan, so test the role in a non-production account first.
• State backend read access: enough to read the state object and acquire the lock. Writing to the backend should be limited to the engineers who apply changes.
• Audit log read access: needed to identify who made a console edit. Without it, step two of the workflow stalls on ownership.

#

Local preparation

Record the versions you are working with, so reviewers can reproduce the output:

1terraform version
2terraform providers
3terraform workspace show

Expected evidence: the Terraform version, the provider versions locked in the dependency lock file, and the workspace name you believe you are in. Paste all three into the change record. Running against the wrong workspace is a realistic way to produce a plan that looks like drift but is not.

#Implementation detail for evidence capture

#

Saving plans as reviewable artefacts

Write plans to files so the reviewed plan is the one applied, and keep a human-readable rendering beside each binary file:

1terraform state pull > state-backup-$(date +%Y%m%d-%H%M%S).json
2terraform plan -out=baseline.tfplan
3terraform plan -refresh-only -out=drift.tfplan
4terraform show -no-color drift.tfplan > drift.txt

Check the flags against your installed version. Plan files can contain sensitive values, so store them with the same protection as state and delete them once the change is closed.

#

Checking the backup before relying on it

A non-empty file is a minimum. Confirm that it parses and that its serial number is recorded:

1jq '.serial, .terraform_version, (.resources | length)' state-backup-*.json

Expected evidence: a serial number, a version and a resource count that roughly matches the estate. If the count is zero for a populated workspace, do not proceed.

#Monitoring the scheduled drift job

A drift pipeline that fails quietly is worse than none, because the team assumes silence means no drift. Treat the job as a monitored service.

• Exit status: run the refresh-only plan with -detailed-exitcode so that a clean result, an error and a detected difference are distinguishable. Confirm the exact exit code meanings in the documentation for your version.
• Heartbeat: alert if no successful run has been recorded within twice the scheduled interval.
• Run duration: record it. A sudden increase often points to API throttling or a new resource type.
• Output retention: keep each rendered result with a timestamp so that you can show when a drifted attribute first appeared.

#Realistic failure symptoms

• Symptom: the refresh-only plan fails with an authorisation message for one resource type. Likely cause: the read-only role lacks a describe permission. Response: add that single permission through normal change control; do not substitute an administrator credential.
• Symptom: the same attribute shows as drifted on every run even after reconciliation. Likely cause: the provider normalises the value differently from your configuration, or another automation rewrites it. Response: compare the audit log entries with the run times before adding any ignore rule.
• Symptom: the job succeeds but lists far fewer resources than expected. Likely cause: wrong workspace, wrong backend key or a partial state. Response: compare the resource count with your last known good backup.
• Symptom: the pipeline hangs waiting for the lock. Likely cause: an interrupted run still holds it. Response: identify the holder from the lock information and follow the lock recovery article rather than forcing it.

#Change-control records

Each reconciliation should leave one record that a reviewer can read without asking questions. Include these fields:

1. Ticket reference and the date of the refresh-only plan.
2. Terraform and provider versions, workspace name and backend location.
3. Filename and checksum of the state backup (sha256sum state-backup-*.json).
4. Per resource: attribute, declared value, observed value, the person who made the edit, the owner who decided, and the decision.
5. Reviewer name for any revert, and the plan file that was approved.
6. Post-change evidence: the clean normal plan and the second refresh-only result.

Where an ignore rule is approved, record its expiry date and the person who will review it, so that it does not become permanent by default.

#Escalation thresholds

Define thresholds before an incident so that the decision does not restThe KBY LexiconRESTREST defines resource-oriented, stateless APIs built on a uniform interface; understanding its constraints clarifies scaling, caching and safe validation of state-changing calls. Read Full Context on one engineer’s judgement at an awkward hour.

• Escalate to the service owner immediately: any drift on network access rules, identity policies, encryption settings or public exposure, and any plan proposing replacement or deletion.
• Escalate to the platform lead the same working day: drift touching more than one service, or drift on resources nobody can attribute.
• Escalate to the backend owner: a state backup that fails validation, an unexplained lock, or a serial number lower than your last recorded value.
• Pause the schedule and raise an incident review: the same resource drifting after two consecutive reconciliations.

Set numeric thresholds for the count of drifted resources per run from your own baseline, not from a generic figure.

#Safe rollback actions

#

After a state-only change

If an applied refresh-only plan recorded values that should not have been accepted, stop all runs for that workspace, take a fresh backup of the current state for forensic comparison, and then restore the earlier copy using the backend’s documented procedure. Confirm with terraform plan that the difference is the one you expect.

#

After an infrastructure change

If a reviewed apply altered a live setting wrongly, prefer a forward correction in code and a second reviewed plan over manual edits. If service impact is immediate, reapply the previous value through the console under an incident record, then reconcile afterwards so that the manual change is captured rather than becoming new drift.

#

Rollback evidence

Record the restored state serial number, the clean plan output and the time at which the service owner confirmed normal behaviour.

Next safe decision: rehearse the sequence on one low-risk resource in a non-production workspace before extending it. For wider context, see Infrastructure as Code on KBY Technologies.