Idempotency Keys: Architecting Exactly-Once Writes

A mobile client on a degraded cellular link fires a POST to /payments, the server debits the account and returns 201 Created, but the response never arrives. The client’s retry logic, doing exactly what it was told to do, fires the same request again. Without a deduplication mechanism at the write boundary, that customer is charged twice. This is not a hypothetical edge case — it is the default behaviour of every HTTP client that implements retries against an at-least-once delivery network, which is to say, all of them. Idempotency keys are the architectural answer, and getting the implementation wrong is far easier than it looks.

The naive fix — “just check if the record already exists before inserting” — collapses the moment you have concurrent retries racing each other, partial failures mid-transaction, or clients reusing a key against a different payload. A production-grade idempotency layer needs a dedicated store, a locking discipline, and a clear contract for what happens when a key is replayed with a different request body. This article covers that contract end to end.

#The Core Problem: At-Least-Once Delivery Meets Non-Idempotent Writes

HTTP does not guarantee exactly-once delivery. TCP resets, load balancer failovers, client timeouts, and proxy retries all mean a logically single request can arrive at your application server zero, one, or multiple times. GET and PUT are naturally idempotent by HTTP semantics — replaying them produces the same end state. POST is not, and most business-critical operations (payments, order creation, inventory reservation) are modelled as POST because they create new resources.

The fix is to shift idempotency from a transport-layer assumption to an application-layer contract. The client generates a unique token — typically a UUIDv4 — and attaches it to every attempt of the same logical operation via an Idempotency-Key header. The server’s job is to guarantee that, no matter how many times a request with that key arrives, the side effect happens exactly once and every caller receives the same response.

#Architectural Breakdown of an Idempotency Layer

A correct implementation has four components sitting in front of the business logic:

• Key extraction and validation — reject requests missing the header on endpoints that mandate it, and scope the key to the caller (tenant ID or API key) to prevent cross-account collisions.
• Request fingerprinting — hash the normalised request body (method, path, and payload) so a replayed key with a mutated payload is detected and rejected rather than silently returning a stale response.
• A dedup store with atomic first-writer-wins semantics — this is the component doing the actual work; it must support a conditional insert that fails loudly on collision.
• Response persistence — the store needs to cache the final HTTP status and body of the first successful execution so replays return an identical response, not just an acknowledgement.

The dedup store is the crux of the design. Whichever backend you choose, it must offer an atomic “insert-if-absent” primitive, because a plain SELECT-then-INSERT pattern reintroduces the exact race condition idempotency keys

#

Choosing a Backend for Idempotency Keys

The decision between Redis, a relational unique constraint, or a managed conditional-write store like DynamoDB depends on your durability and latency requirements. This is one of the more consequential architectural patterns decisions in the whole design, because the backend choice dictates your failure envelope during outages.

For payment and ledger systems, the relational or DynamoDB path is worth the extra latency because it survives a Redis restart without silently forgetting keys mid-processing. For lower-stakes idempotency, such as deduplicating webhook deliveries, Redis’s speed usually wins the trade-off.

#Implementation Logic

The request lifecycle for a correctly implemented idempotency key middleware looks like this:

The intermediate processing state is the piece most naive implementations skip, and it is what prevents two concurrent retries from both executing the business logic before either has finished writing the cached response.

#

Schema and Middleware Reference

A relational schema for the dedup store needs a composite uniqueness constraint scoped by tenant, plus a fingerprint column for payload verification:

1CREATE TABLE idempotency_keys (
2 tenant_id UUID NOT NULL,
3 idempotency_key TEXT NOT NULL,
4 request_fingerprint TEXT NOT NULL,
5 status TEXT NOT NULL DEFAULT 'processing',
6 response_status INT,
7 response_body JSONB,
8 created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
9 expires_at TIMESTAMPTZ NOT NULL,
10 PRIMARY KEY (tenant_id, idempotency_key)
11);
12
13CREATE INDEX idx_idem_expiry ON idempotency_keys (expires_at);

The middleware logic, expressed as pseudocode, enforces the fingerprint check before doing anything else:

1async function idempotencyMiddleware(req, res, next) {
2 const key = req.headers['idempotency-key'];
3 if (!key) return res.status(400).json({ error: 'Idempotency-Key header required' });
4
5 const fingerprint = sha256(`${req.method}:${req.path}:${canonicalize(req.body)}`);
6
7 try {
8 await db.query(
9 `INSERT INTO idempotency_keys (tenant_id, idempotency_key, request_fingerprint, expires_at)
10 VALUES ($1, $2, $3, now() + interval '24 hours')`,
11 [req.tenantId, key, fingerprint]
12 );
13 // First writer wins — proceed to business logic
14 req.idempotencyKey = key;
15 next();
16 } catch (err) {
17 if (err.code === '23505') { // unique_violation
18 const row = await db.query(
19 `SELECT * FROM idempotency_keys WHERE tenant_id=$1 AND idempotency_key=$2`,
20 [req.tenantId, key]
21 );
22 if (row.request_fingerprint !== fingerprint) {
23 return res.status(422).json({ error: 'Idempotency-Key reused with different payload' });
24 }
25 if (row.status === 'processing') {
26 return res.status(409).json({ error: 'Request already in flight' });
27 }
28 return res.status(row.response_status).json(row.response_body);
29 }
30 throw err;
31 }
32}

This pattern is close to the one described in the IETF draft specification for the Idempotency-Key HTTP header, which standardises header naming and 422/409 status semantics across vendor implementations.

#Failure Modes and Edge Cases

Idempotency keys introduce their own failure surface, and it is worth enumerating the ones that actually bite in production:

• Orphaned “processing” rows — if the application server crashes after the conditional insert but before the business logic completes, the key is stuck in processing forever, permanently blocking retries. Mitigate with a processing-state timeout (e.g. 30 seconds) after which the row is eligible for reclaim by a subsequent request.
• Fingerprint drift from non-deterministic serialisation — JSON key ordering, floating-point formatting, or optional-field omission can cause the same logical payload to hash differently across retries. Canonicalise the payload (sorted keys, fixed decimal precision) before hashing.
• TTL expiry racing a slow downstream call — if the idempotency window is shorter than your p99 processing latency, a legitimate retry can be treated as a brand-new request after the original row expires, defeating the entire mechanism. Size the TTL against your slowest dependency, not your median.
• Partial side effects on multi-step writes — if order creation and payment capture are two separate downstream calls, a crash between them leaves an inconsistent state that a naive idempotency check will not detect on replay. This requires a saga or outbox pattern layered on top of the idempotency key, not as a substitute for it.
• Cross-region replication lag — in an active-active deployment, a key inserted in one region may not yet be visible in another when a retry lands on a different node, producing duplicate execution despite a correctly implemented single-region check.

#Scaling and Security Trade-offs

Once idempotency keys are load-bearing infrastructure, several trade-offs become operationally significant:

• Storage growth — a high-throughput API generating millions of keys daily needs an aggressive TTL and a partitioned table (or Redis key expiry) to avoid unbounded growth; unindexed expiry scans on a multi-hundred-million-row table will degrade write latency across the whole service.
• Hashing cost on large payloads — fingerprinting a multi-megabyte upload body on every request adds measurable CPU overhead; hash a truncated canonical subset (e.g. business-relevant fields only) rather than the full serialised body where payloads are large.
• Key namespace isolation — scoping keys by tenant/API-key prevents one customer’s idempotency key colliding with another’s, but also means a compromised API key can only replay that tenant’s own requests, limiting the blast radius of a leaked key.
• Replay as an attack vector — an idempotency key does not authenticate a request; an attacker who captures a valid key and payload can replay it within the TTL window to retrieve the cached response, which may leak sensitive fields to an unauthorised session if authorisation is checked only on the first execution.
• Cross-region consistency cost — enforcing a global uniqueness guarantee across regions typically requires either a single-region-authoritative write path (adding latency for distant clients) or a consensus-backed store like DynamoDB global tables, which trades stronger consistency for higher per-write cost and eventual-consistency windows on the TTL attribute itself.

None of these trade-offs argue against using idempotency keys — they argue for treating the dedup store with the same rigour as the primary database it is protecting. A payments team that builds idempotency keys as an afterthought on top of Redis with no fingerprint check and no processing-state timeout has built a system that looks correct in the happy path and fails exactly when the network conditions that necessitated the mechanism in the first place — timeouts, retries, partial outages — actually occur.