·11 min read
Idempotency Is a Scaling Problem, Not a Correctness Footnote
Double charges, phantom inventory, ghost orders: all the same bug. The operation was safe to run once and your infrastructure ran it twice.
Backend / Architecture / Distributed Systems / E-commerce / PostgreSQL / Node.js
On one server with one client, a POST request either succeeds or fails, and you can pretend the world is simple. Add a load balancer, a mobile client on a flaky connection, a queue between services, or a payment provider whose webhook fires twice, and the pretence breaks: every operation that changes state will, given enough traffic, execute more than once.
The symptoms look like application bugs. A customer is charged twice. An item appears in the cart three times after one tap. An order confirmation email goes out, followed by a second identical one sixty seconds later. Inventory drops below zero. Each one is investigated as a one-off, patched with a guard clause, and then it happens again in a different shape. The underlying cause is always the same: an operation that was designed to run once was not designed to be safe to run twice.
Nothing runs exactly once
The network between any two computers offers two guarantees: at-most-once (send and do not retry, you might lose it) and at-least-once (retry until acknowledged, you might duplicate it). Exactly-once delivery does not exist at the transport level. Every system that appears to offer it is implementing at-least-once delivery with deduplication at the receiver, which is idempotency by another name.
The retries come from everywhere:
- The browser or mobile app retries a POST because the response was lost, even though the server processed it.
- A load balancer retries against a second instance because the first returned a 502 mid-response.
- A message broker delivers the same message twice after a consumer crash before acknowledgement.
- A payment webhook fires again because your endpoint returned a 200 too slowly and the provider treated it as a timeout.
- A user taps "Place Order" twice because the button did not disable fast enough.
Disabling the button is a UI courtesy, not a safety mechanism. The safety mechanism is the server treating the second request as a replay of the first and returning the same result without doing the work again.
The shopping cart that double-adds
The simplest example is adding an item to a cart. The naive implementation inserts a row or increments a quantity:
// The version that breaks under retry
app.post("/cart/items", async (req, res) => {
const { cartId, productId, quantity } = req.body;
await db.query(
`INSERT INTO cart_items (cart_id, product_id, quantity)
VALUES ($1, $2, $3)
ON CONFLICT (cart_id, product_id)
DO UPDATE SET quantity = cart_items.quantity + $3`,
[cartId, productId, quantity],
);
res.json({ ok: true });
});
If the client sends this request once, the cart gets one unit. If the response is lost and the client retries, the cart gets two. The ON CONFLICT clause, which looks like it handles duplicates, makes the problem worse: it guarantees that every retry increments.
The fix is to make the operation describe the desired state rather than a delta:
// Idempotent: describes the target state, not a delta
app.post("/cart/items", async (req, res) => {
const { cartId, productId, quantity } = req.body;
await db.query(
`INSERT INTO cart_items (cart_id, product_id, quantity)
VALUES ($1, $2, $3)
ON CONFLICT (cart_id, product_id)
DO UPDATE SET quantity = $3`,
[cartId, productId, quantity],
);
res.json({ ok: true });
});
Now the second request sets the quantity to 1 again instead of incrementing it to 2. The result is the same regardless of how many times the request arrives. That is the core of idempotency: the operation produces the same outcome whether it runs once or ten times.
Not every operation can be converted to a PUT-style overwrite. "Add one more of this item" is inherently a delta, and the intent matters. The answer for those cases is an idempotency key.
The idempotency key
An idempotency key is a unique string the client generates before sending the request and includes as a header. The server uses it to recognise a retry: if it has seen this key before, it returns the stored response without executing the operation again.
// Client side: generate before the first attempt
const idempotencyKey = crypto.randomUUID();
// Every retry sends the same key
await fetch("/api/orders", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify(order),
});
The key must be generated by the client, not the server. The whole point is that the client knows whether it is sending a new request or re-sending a failed one. A UUID per user action is the simplest choice. For operations where the natural identity is obvious (a specific checkout of a specific cart) a deterministic key works too: checkout:${cartId}:${cartVersion} encodes both what is being done and the state it was done against.
Deterministic keys have a useful property: they prevent the operation from running twice even across different sessions or devices, which a random UUID does not. A user who opens the checkout on their phone and their laptop simultaneously gets one order, not two.
A ledger in Postgres
The server needs to store which keys it has seen and what it returned. This is a small table:
CREATE TABLE idempotency_ledger (
key text PRIMARY KEY,
route text NOT NULL,
status_code smallint,
response jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX idx_idempotency_created
ON idempotency_ledger (created_at);
The lookup and the business logic run in the same transaction. That is the part that matters: if the key check and the operation are not atomic, a crash between the two either loses the key (and the retry executes twice) or records the key without completing the work (and the retry returns a stale result for an operation that never finished).
async function withIdempotency<T>(
key: string,
route: string,
work: () => Promise<{ status: number; body: T }>,
): Promise<{ status: number; body: T }> {
return db.transaction(async (tx) => {
const existing = await tx.query(
`INSERT INTO idempotency_ledger (key, route)
VALUES ($1, $2)
ON CONFLICT (key) DO NOTHING
RETURNING key`,
[key, route],
);
if (existing.rowCount === 0) {
// Key already exists
const stored = await tx.query(
`SELECT status_code, response
FROM idempotency_ledger WHERE key = $1`,
[key],
);
const row = stored.rows[0];
if (!row.status_code) {
// Previous attempt started but never finished:
// let this attempt run. A reaper cleans rows
// stuck longer than 60 seconds.
return work();
}
return { status: row.status_code, body: row.response as T };
}
// First time seeing this key
const result = await work();
await tx.query(
`UPDATE idempotency_ledger
SET status_code = $1, response = $2
WHERE key = $3`,
[result.status, JSON.stringify(result.body), key],
);
return result;
});
}
The incomplete-row case matters. If the server crashes after claiming the key but before finishing the work, the next retry sees a row with no response. Treating that as "already done" would return nothing forever. Instead, the retry re-executes the work, which is safe because the original attempt never completed. A background reaper deletes rows older than a minute that have no response, which handles the case where the retry never arrives.
Twenty-four hours is a reasonable TTL for the ledger. A retry that arrives a day later is not a retry. It is a new request wearing old clothes, and treating it as a replay would be more confusing than processing it fresh. Run a nightly DELETE or use Postgres partitioning if the volume justifies it.
Checkout, payment, and the two-write problem
Checkout is where the stakes rise. The operation touches at least two systems (your database and the payment provider) and neither knows about the other. The classic failure: you charge the card, crash before recording the order, and the retry charges the card again.
The payment provider solves their half of this with their own idempotency key. Stripe accepts an Idempotency-Key header on every mutating request: two calls with the same key produce one charge. The discipline is to derive the payment idempotency key from your idempotency key, so the mapping is deterministic:
async function processCheckout(
idempotencyKey: string,
cart: Cart,
): Promise<Order> {
return withIdempotency(idempotencyKey, "checkout", async () => {
// 1. Create the order in your database (pending)
const order = await createOrder(cart, "pending");
// 2. Charge the card with a derived key
const paymentKey = `payment:${idempotencyKey}`;
const charge = await stripe.paymentIntents.create(
{
amount: cart.totalCents,
currency: "usd",
metadata: { orderId: order.id },
},
{ idempotencyKey: paymentKey },
);
// 3. Record the outcome
await updateOrder(order.id, {
status:
charge.status === "succeeded"
? "confirmed"
: "payment_failed",
stripePaymentId: charge.id,
});
return {
status: 201,
body: {
orderId: order.id,
paymentStatus: charge.status,
},
};
});
}
Step 1 creates the order as pending before charging. If the charge fails or the process crashes, the order exists in a state that a reconciliation job can clean up. A pending order is visible and fixable; an unrecorded charge requires digging through the provider's dashboard.
The order of operations matters: create the record in the system you control first, then call the external system. If the external call fails, you have a record to reconcile against. If you call the external system first and crash before recording, you have nothing.
A webhook from the payment provider is the belt to the suspenders. Stripe sends a payment_intent.succeeded event that your webhook handler uses to confirm the order independently. The webhook itself is at-least-once (Stripe retries for days), so the handler must be idempotent too. Update the order to confirmed only if it is still pending, and ignore the event if it is already confirmed:
UPDATE orders SET status = 'confirmed', confirmed_at = now()
WHERE stripe_payment_id = $1 AND status = 'pending';
-- Rows affected: 1 on first delivery, 0 on every replay
Inventory holds that survive retries
Inventory is the other state that breaks under duplicate requests. A customer adds an item to a cart, and you want to hold that stock so it is not sold to someone else before checkout completes. The naive approach decrements inventory on add-to-cart:
-- Breaks under retry: each attempt decrements again
UPDATE products SET stock = stock - 1
WHERE id = $1 AND stock > 0;
Two retries, two decrements, one item. The customer sees one item in their cart and you have reserved two. Multiply by a thousand concurrent sessions and your inventory is fiction.
The fix is to model the hold as an entity, not a delta:
CREATE TABLE inventory_holds (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
product_id uuid NOT NULL REFERENCES products(id),
cart_id uuid NOT NULL,
quantity int NOT NULL,
held_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz NOT NULL
DEFAULT now() + interval '15 minutes',
released boolean NOT NULL DEFAULT false,
UNIQUE (cart_id, product_id)
);
The unique constraint on (cart_id, product_id) makes the hold creation naturally idempotent: the second attempt finds an existing hold and returns without touching stock. Available inventory is computed from the product's stock minus active, unexpired holds, which means holds expire on their own if the cart is abandoned. A background job releases expired holds, or the checkout process converts them into a permanent decrement.
async function holdInventory(
cartId: string,
productId: string,
quantity: number,
): Promise<"held" | "insufficient" | "already_held"> {
return db.transaction(async (tx) => {
const existing = await tx.query(
`SELECT id FROM inventory_holds
WHERE cart_id = $1
AND product_id = $2
AND NOT released`,
[cartId, productId],
);
if (existing.rowCount > 0) return "already_held";
const available = await tx.query(
`SELECT p.stock - COALESCE(SUM(h.quantity), 0)
AS available
FROM products p
LEFT JOIN inventory_holds h
ON h.product_id = p.id
AND NOT h.released
AND h.expires_at > now()
WHERE p.id = $1
GROUP BY p.id`,
[productId],
);
if (available.rows[0].available < quantity) {
return "insufficient";
}
await tx.query(
`INSERT INTO inventory_holds
(product_id, cart_id, quantity)
VALUES ($1, $2, $3)`,
[productId, cartId, quantity],
);
return "held";
});
}
This is more machinery than a simple decrement, and it is the right amount. An inventory system that cannot survive a retry is one that will, at some traffic level, sell what it does not have or fail to sell what it does.
Idempotency behind a queue
When the operation moves behind a message queue (order fulfilment, sending a confirmation email, updating a search index) the idempotency moves with it. The Kafka post covered the principle: at-least-once delivery plus an idempotent handler is the realistic design, because exactly-once delivery does not exist and at-most-once loses messages.
The mechanism is the same ledger pattern, keyed on the message's unique ID:
async function handleOrderConfirmation(msg: QueueMessage) {
const claimed = await db.query(
`INSERT INTO processed_messages
(message_id, queue, claimed_at)
VALUES ($1, $2, now())
ON CONFLICT (message_id) DO NOTHING
RETURNING message_id`,
[msg.id, "order_confirmation"],
);
if (claimed.rowCount === 0) return; // already processed
await sendConfirmationEmail(msg.payload);
await db.query(
`UPDATE processed_messages
SET completed_at = now()
WHERE message_id = $1`,
[msg.id],
);
}
The email case is worth calling out, because it is the one that users notice most visibly. A customer receiving two identical order confirmations does not think "oh, the queue delivered twice." They think the system is broken, or they think they were charged twice and open a support ticket. The deduplication table is cheap; the support tickets are not.
For the event-sourced notification pipeline described in an earlier post, the event ID is the natural idempotency key. Replaying the event log (which is the whole point of having one) is only safe because every consumer is idempotent on event ID. Without that property, replay is replay of bugs.
When you do not need this
Idempotency has a cost: a table, a lookup on every request, a transaction that wraps more than it otherwise would, and a TTL policy to maintain. It is the right investment in specific circumstances and overhead in others.
- Read-only operations. GET requests are idempotent by definition. No ledger needed, no key needed. If your GET has side effects, that is the bug, not the missing idempotency.
- Naturally idempotent writes. An upsert that sets a value rather than incrementing it is already safe to retry. The cart example above (
SET quantity = $3rather thanquantity + $3) is the pattern. If every write in the system describes a target state, you do not need a separate ledger. - Internal calls you fully control. If both sides are your code, in the same data centre, with no load balancer or queue between them, and you can guarantee one call per operation by construction (a synchronous function call, not an HTTP request), the ledger is overhead. The moment a network boundary, a retry policy, or a queue enters the picture, this exception stops applying.
- Low-stakes operations. A log line written twice is not a problem. An analytics event counted twice is a rounding error in most systems. Reserve the machinery for operations where a duplicate has a cost: money moved, inventory changed, a user notified, a record created that triggers downstream work.
The short version
The network does not deliver messages exactly once, so any operation that changes state will eventually execute more than once. For shopping carts, make writes describe the target state rather than a delta. For operations that are inherently non-idempotent (charges, sends, anything with a side effect outside your database), the mechanism is a client-generated idempotency key checked in a Postgres ledger inside the same transaction as the business logic. For checkout, create the order before charging, derive the payment provider's idempotency key from yours, and reconcile via webhook. For inventory, model holds as entities with a unique constraint rather than as decrements. Behind a queue, deduplicate on message ID. The principle is the same in every case: the second request should be indistinguishable from a successful response to the first, and the cost of building that property is less than the cost of the refunds, support tickets, and data repairs that arrive without it.