Level-Triggered Reconciliation

A control loop pattern where a controller repeatedly diffs observed state against desired state and acts on the current diff, rather than reacting to individual change events.

Practical example

A Kubernetes Deployment controller does not track 'pod X was just created'; on every reconcile it counts current ready pods, compares against spec.replicas, and creates or deletes pods to close the gap, whether triggered by a watch event or a periodic resync.

Level-triggered reconciliation is the operating model underlying Kubernetes controllers, most operators, and many autoscalers. A controller runs a loop: observe(actualState) -> diff(desiredState, actualState) -> act(). Critically, the loop does not depend on remembering what triggered it. Whether invoked because of a watch event, a periodic resync, or a manual requeue, the controller re-derives the full diff from the current state snapshot every time. This is the opposite of an edge-triggered model, where a handler fires once per discrete transition (e.g., ‘pod created’, ‘pod deleted’) and must track transition history to remain correct.

The architectural payoff is idempotency under message loss. In an edge-triggered system, a missed ‘delete’ event leaves a stale resource forever; a duplicate ‘create’ event can double-provision. In a level-triggered system, the next reconciliation pass simply recomputes the diff and self-corrects, because the loop treats the desired-state object as the single source of truth and the cluster state as ephemeral, disposable evidence. This is why Kubernetes controllers pair watch-based triggers (for low latency) with a periodic full resync (for correctness): the watch is an optimization, the resync is the correctness guarantee.

Requeue semantics: controllers use exponential backoff requeueing on transient errors rather than crashing, because level-triggering tolerates delayed convergence but not silent divergence.
Read-your-writes hazards: if the controller reads from a stale informer cache immediately after writing, it may recompute a diff against outdated state and issue a redundant or conflicting action; optimistic concurrency (resourceVersion checks) mitigates this.
Thundering herd on resync: a large full resync interval across thousands of objects can spike API server load; jittered resync periods are used to spread this.
Non-idempotent side effects: external API calls (e.g., cloud LB provisioning) inside a reconcile function must themselves be idempotent, or repeated reconciliation passes will duplicate side effects even though the control loop itself is correct.

The failure mode engineers most commonly introduce is smuggling edge-triggered logic into a level-triggered controller — e.g., storing ‘has this been processed’ flags in memory instead of in the object’s status subresource. This breaks correctness on controller restart, because in-memory state is lost while the reconciler is re-invoked against a resource that now appears ‘new’ but was actually already handled, or vice versa. Status must be persisted as observable state, not inferred from event history.