Designing Compensating Sagas for Microservices

A distributed transaction that spans four microservices and a payment gateway cannot rely on a two-phase commit coordinator without introducing a synchronous lock across services that were explicitly decoupled to avoid synchronous coupling. The moment you attempt XA transactions across service boundaries owned by different teams, you reintroduce the exact failure domain that microservice decomposition was meant to eliminate: one slow participant blocks every other participant’s resources until the coordinator times out. This is the precise gap the saga orchestration pattern fills — it replaces atomic commit with a sequence of local transactions, each paired with a compensating action that can undo its effect if a later step fails.

This article covers the orchestration variant specifically, not choreography, because orchestration is the version that survives an audit. When a regulator or an incident review asks “what happened to order #48291,” you need a single log you can query, not a forensic reconstruction of six services’ event streams.

#Problem Statement: Why Local Transactions Alone Are Insufficient

Consider an order-fulfilment flow: reserve inventory, charge payment, schedule shipping, notify the customer. Each step is a local ACID transaction inside its own service and database. The failure mode that matters is not the happy path — it is what happens when step three (shipping) fails after step two (payment) has already succeeded. Without a coordinating mechanism, the payment has been captured but no shipment exists. The system is now in a state that violates business invariants, and there is no database-level rollback available because the transaction boundary was never shared.

The saga orchestration pattern formalises the response: every forward step must have a corresponding compensating transaction defined before the saga executes. If step three fails, the orchestrator does not attempt to retry indefinitely — it walks backwards through the completed steps, invoking their compensations in reverse order. This is not rollback in the ACID sense; it is a business-level correction (refund the payment, release the inventory hold) rather than a storage-engine undo log.

#Architectural Breakdown: Orchestration vs Choreography

Choreography-based sagas rely on each service reacting to events emitted by its predecessor, with no central authority. This scales well for two or three steps but becomes unmanageable past five or six, because the “transaction logic” is smeared across every service’s event handlers with no single place to read the full workflow. Orchestration centralises that logic into a dedicated coordinator — typically a state machine — that issues commands and listens for outcomes.

The orchestrator’s single-point-of-failure risk is manageable if the orchestrator itself is stateless between invocations, persisting its progress to a durable store after every transition rather than holding state in memory. This is precisely the design AWS Step Functions and Azure Durable Functions converge on, and it’s worth reviewing Microsoft’s own reference implementation of the saga pattern in the Azure Architecture Center before building a bespoke coordinator, since most of the edge cases around timeout handling are already documented there.

#Implementation Logic: Building the State Machine

The orchestrator needs three components: a step definition table, a persistence layer for in-flight saga state, and a dispatcher that maps step outcomes to either the next forward step or the previous compensation. The critical design decision is that every command sent to a participant service must carry an idempotency key

1{
2  "sagaId": "order-48291",
3  "currentStep": 2,
4  "status": "COMPENSATING",
5  "steps": [
6    { "name": "reserveInventory", "status": "COMPLETED", "compensatedAt": null },
7    { "name": "chargePayment", "status": "COMPLETED", "compensatedAt": null },
8    { "name": "scheduleShipping", "status": "FAILED", "reason": "CARRIER_TIMEOUT" },
9    { "name": "notifyCustomer", "status": "NOT_STARTED" }
10  ],
11  "idempotencyKeyPrefix": "order-48291-step-"
12}

This JSON document is the durable record — persisted before dispatch, not after — so a crash between sending the command and receiving the response leaves an unambiguous “in-flight” marker the recovery process can pick up. Recovery logic on orchestrator restart must scan for any saga in a non-terminal state older than its configured timeout and resume either forward execution or compensation, never assume the last known status is still accurate.

#

Compensation Handler Structure

Each participant service exposes both a forward endpoint and a compensation endpoint, and the compensation endpoint must be safe to call even if the forward action never actually completed (a common scenario when the orchestrator times out waiting for a response that was in fact successful but lost in transit).

1async function compensateChargePayment(sagaId: string, idempotencyKey: string) {
2  const existing = await paymentLedger.findByIdempotencyKey(idempotencyKey);
3
4  if (!existing || existing.status === 'REFUNDED') {
5    // Nothing to compensate u2014 return success to unblock the saga
6    return { status: 'SKIPPED', reason: 'no_matching_charge' };
7  }
8
9  const refund = await paymentGateway.refund(existing.chargeId, {
10    idempotencyKey: `${idempotencyKey}-refund`,
11  });
12
13  await paymentLedger.markRefunded(existing.id, refund.id);
14  return { status: 'COMPENSATED', refundId: refund.id };
15}

Note the second idempotency key appended with -refund. Reusing the original key for the compensating call risks the payment gateway treating it as a duplicate of the original charge request rather than a distinct refund operation — a subtle bug that only surfaces under retry storms.

#

Persistence Schema for the Saga Ledger

1CREATE TABLE saga_instances (
2  saga_id UUID PRIMARY KEY,
3  saga_type VARCHAR(64) NOT NULL,
4  current_step SMALLINT NOT NULL,
5  status VARCHAR(24) NOT NULL, -- RUNNING, COMPENSATING, COMPLETED, FAILED
6  payload JSONB NOT NULL,
7  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
8  version INT NOT NULL DEFAULT 1
9);
10
11CREATE UNIQUE INDEX idx_saga_step_idempotency
12  ON saga_step_log (saga_id, step_index, idempotency_key);

The version column enforces optimistic concurrency control

#Sequencing the Compensation Path

This diagram makes explicit what a purely textual description obscures: compensation runs in reverse chronological order, not in parallel. Running compensations concurrently is tempting for latency reasons but introduces ordering hazards when a later step’s compensation logically depends on an earlier step still holding its resource — releasing inventory before confirming the refund, for instance, could allow the same unit to be sold twice while a refund is still pending.

#Failure Modes and Edge Cases

The saga orchestration pattern does not eliminate partial failure; it relocates it to a place where it can be reasoned about. Three edge cases dominate production incidents:

Compensation failure itself. If compensateChargePayment fails — the payment gateway is down — the orchestrator cannot simply mark the saga FAILED and move on, because the customer has now been charged with no shipment and no refund. The correct behaviour is to move the saga into a COMPENSATION_FAILED state that triggers a page to an on-call operator, with automatic retry using exponential backoff capped at a bounded number of attempts before escalation. Never let a compensation step fail silently.

Non-idempotent side effects. Sending a customer notification email is not naturally reversible — you cannot un-send an email. Steps like this are typically ordered last in the forward path specifically so they never require compensation, since by the time they execute, every preceding step has already succeeded durably.

Orchestrator version skew during deployment. If you deploy a new orchestrator version that changes the step sequence while sagas from the old version are still in-flight, the state machine definition used to resume a saga must match the definition it started under. Persisting a schemaVersion field alongside the saga instance, and keeping the old step-handler code path available until every in-flight instance of that version drains, avoids resuming a saga against logic it was never designed for.

#Scaling and Security Trade-offs

Treating the saga orchestration pattern as a drop-in replacement for distributed locking ignores the operational cost of running a highly available state machine. These trade-offs are worth stating explicitly when proposing the pattern within broader architectural patterns discussions at design review:

• Throughput ceiling: every step transition is a synchronous write to the saga ledger; expect the ledger’s write throughput, not the participant services, to become the bottleneck above roughly 2,000-3,000 concurrent in-flight sagas per orchestrator shard, depending on the underlying database’s commit latency.
• Horizontal scaling of the orchestrator: partition saga instances by sagaId hash across orchestrator shards rather than running a single logical coordinator, but ensure the version-controlled write pattern described above holds per shard to avoid split-brain advancement of the same instance.
• Security surface of compensation endpoints: compensation handlers are frequently under-tested compared to forward handlers and become an attractive target for replay attacks, since a compensating refund call executed twice against a poorly guarded idempotency check produces a real financial loss; apply the same authentication and rate-limiting controls to compensation routes as to the forward-path routes.
• Audit and compliance cost: the centralised ledger is an asset for SOX/PCI audits but becomes a single high-value target; encrypt the payload column at rest and restrict direct database access to the orchestrator service account only, routing all human inspection through a read-only reporting view.
• Latency versus consistency: orchestration adds one network round-trip per step compared to choreography’s fire-and-forget events; for workflows under a 200ms end-to-end latency budget, this overhead is frequently the deciding factor against orchestration, and a hybrid model — choreography for the fast path, orchestration only for the compensation path — is worth evaluating.

None of this makes the saga orchestration pattern a universal substitute for transactional guarantees. It is a deliberate trade of strong consistency for availability and service autonomy, and the compensating logic it demands only pays off when your team treats compensation handlers with the same rigour, test coverage and security review as the forward-path business logic they are meant to undo.