The failure that motivates all of this
Consider the simplest possible bug. You call POST /v1/payout, we broadcast the transaction, and then your HTTP client's read timeout fires before our 200 reaches you. You don't know whether the payout happened. The safe-looking move — retry — is exactly the dangerous one: without idempotency, the second request is a second, independent payout. On a card network you could reverse it. On-chain, once it's confirmed, it's gone.
The fix is an old one from payment APIs: attach a client-chosen key to the request, and make the server promise that two requests carrying the same key produce one effect. Retry as many times as you like; you get the same payout back every time. What follows is how that promise is actually kept in our code, per surface — because "we support idempotency keys" is a claim that lives or dies in the race conditions.
Payouts: idempotency_key, scoped per merchant
POST /v1/payout accepts an optional idempotency_key (8–128 chars; also honored as the X-Idempotency-Key header). We store it on the Payout row and enforce a unique constraint. The scoping detail matters: uniqueness is per merchant, not global. Your invoice-4417 and another merchant's invoice-4417 are different keys — you never collide with, or leak into, someone else's namespace. Pick something stable and meaningful from your side: the order id, the settlement batch line, a UUID you persist before the first call.
The lookup happens twice on purpose, and that redundancy is the whole game. We check for an existing payout with your key before doing any work — the cheap common case. But two concurrent retries can both pass that first check, so the real guard is inside a per-merchant lock, where we re-check after acquiring it. And even that is backstopped by the database: the row insert can still lose a race, so a P2002 unique-constraint violation is caught and resolved by returning the winner's row. Three layers — pre-check, in-lock re-check, unique-constraint catch — all converging on the same answer: one payout, returned as idempotent.
// Same key, twice — the second call is a no-op that returns the first payout
if (params.idempotencyKey) {
const existing = await prisma.payout.findFirst({
where: { idempotencyKey: params.idempotencyKey, merchantId }, // per-merchant scope
});
if (existing) return { payout: existing, idempotent: true };
}
// ...acquire per-merchant lock, re-check, then debit + create in one transaction.
// A losing P2002 race is caught and resolved to the winner's row.One thing the key does not do: bypass the balance debit safety. The debit and the row creation commit in a single database transaction, so a crash between them rolls back cleanly — you're never debited for a payout that doesn't exist.
Batch payouts: external_id per line
Mass payouts have the same problem, one level down: it's not the batch you want to dedupe, it's each recipient line. So the batch endpoint takes an external_id per item, and that maps straight onto the very same Payout.idempotencyKey column with the same unique guarantee. Resubmit a batch after a partial failure and only the lines that didn't already land get created; the ones that did come back as their existing rows. We also reject duplicate external_id values within a single batch up front, so a copy-paste mistake is a clean 400 rather than a half-applied run. The mechanics of batching, netting, and per-line status are in the mass payouts guide.
Transfers: the same discipline, internal rails
Internal transfers between balances carry an idempotency_key too (1–64 chars, or the X-Idempotency-Key header), with the same per-merchant scoping. The point of repeating the pattern everywhere money moves is that you shouldn't have to remember which endpoints are safe to retry. They all are — provided you send a key.
Why "retry blindly" is genuinely safe here
The idempotency key protects the creation of a payout. A second, deeper guard protects the broadcast. Before the worker signs anything, it NX-sets a Redis sentinel keyed on the payout id. Only the first attempt wins that set; any retry finds it present and refuses to re-broadcast. That's what lets us run the queue with attempts: 1 and still survive worker crashes — the sentinel, not blind auto-retry, decides what may go on-chain.
// Worker — set BEFORE signing; only the first attempt may broadcast
const canBroadcast = await redis.set(`payout:broadcast:${payout.id}`, "1", "EX", 86400, "NX");
if (!canBroadcast) {
// A previous attempt already crossed the signing line — do not re-broadcast.
return; // reconcile status against the chain instead
}The companion rule is about failures, and it's deliberately conservative: we never re-credit an ambiguous state. A payout that failed cleanly before the broadcast line rolls its status to FAILED and re-credits your balance in one atomic step — safe, because nothing left the wallet. But a payout that was submitted and then lost its confirmation check, or whose post-broadcast database write failed, is left in place and flagged for a human to reconcile against the chain. Re-crediting there would be the double-pay. We would rather page an operator than hand you back money that already moved. This is the same "the chain is the source of truth" stance behind payment finality.
Webhooks: the deterministic event_id is our half of the deal
Everything above is about requests you send us. Webhooks flip the direction, and here the idempotency key isn't yours to choose — it's ours to guarantee. Every outbound webhook carries an event_id, and it is deterministic: it's derived from the event type and the entity id, so every retry of the same logical event — the inner 5-attempt delivery loop, a tick-level replay after a rollback, a manual redispatch — carries the identical id. You dedupe on it and a re-delivery becomes a no-op.
The derivation is worth stating exactly, because "some hash" isn't a contract. We compute SHA-256(event_type + ":" + entity_id), take the first 16 bytes (128 bits), and encode them as 26 characters of Crockford base32 (alphabet 0123456789ABCDEFGHJKMNPQRSTVWXYZ — no I, L, O, U). Same event type, same entity, same id — forever, on every machine, with no shared state or database sequence involved. It's a pure function, which is why we can promise stability across retries that might be separated by process restarts.
// How we derive it — a pure function of (event_type, entity_id)
const h = crypto.createHash("sha256").update(eventType).update(":").update(entityId).digest();
// ...take the first 16 bytes, encode as 26 chars of Crockford base32 → e.g. "payment.paid" + payment id
// Your side:
if (await seen(event.event_id)) return res.sendStatus(200); // already applied
await applyInTransaction(event);
await markSeen(event.event_id);Note what determinism buys you beyond simple retry-dedup: because the id is a function of the event, not a random token minted at send time, two independent deliveries of the same state collapse to one id on your side automatically. The full consumer checklist — HMAC verification, ordering, reconciliation — lives in reliable payment webhooks.
Where our job ends and yours begins
The honest boundary: the deterministic event_id is ours — you get it for free, we compute it the same way every time, and you can trust it. The idempotency_key on payouts, transfers, and batch lines is yours — we enforce uniqueness and resolve every race behind it, but we can't invent a stable key for a request you haven't sent yet. If you omit it, a retry is a new payout, and that's on the caller, not the platform.
So the whole rule reduces to one habit: generate an idempotency key before the first attempt, persist it, and send it on every retry. Do that and the ugliest failure in payments — the timeout where you don't know if the money moved — becomes boring. You retry, you get the same result, and nobody gets charged or paid twice. Manage keys and inspect deliveries from the payouts dashboard.