Design a Payment System: Ledgers, Idempotency, and Sagas

Money systems have one property most designs do not: a bug is not a stale cache, it is a wrong number in someone's bank account. That single constraint drives every interesting decision when you design a payment platform like Stripe. Duplicate requests must not double-charge. A crash mid-flight must not leave money half-moved. Every cent that enters the system must trace back to a source you can name months later during an audit.

So a strong payment-system answer is less about boxes on a whiteboard and more about invariants: what stays true no matter which retry fires, which pod restarts, or which webhook arrives out of order. The famous techniques — idempotency keys, double-entry bookkeeping, the saga pattern — all exist to protect one of those invariants.

Below is the design built the way an interviewer expects: requirements first, then real capacity numbers, then architecture, API, data model, and the deep dives where candidates either shine or fall apart.

What a payment system must actually do#

Strip away the branding and a payment platform does four things: it accepts a charge request, moves money through a card network via a Payment Service Provider (PSP), records that movement in a ledger, and pays out the merchant later. Everything else — refunds, disputes, partial captures, payouts — is a variation on those.

The one distinction worth internalizing early is authorization versus capture. An authorization places a hold on the cardholder's available balance: the issuer reserves the funds but no money moves yet. Capture is the step that actually pulls those funds and starts settlement. They are deliberately split so a merchant can authorize at checkout, run fraud review, and only capture when the item ships. Retail ships-then-charges, travel overbooks and adjusts, and marketplaces do partial captures across split shipments. Card networks also reward late capture with lower interchange, so the two-phase model is a cost lever, not just a convenience.

The non-functional requirements are where payments diverge from a typical CRUD service. Correctness beats availability on the write path — it is better to reject a charge than to record a phantom one. Durability is non-negotiable: an acknowledged charge must survive a datacenter loss. And the whole system lives inside a compliance boundary that shapes the architecture before you draw a single service.

How much does the ledger actually grow?#

Numbers turn hand-waving into engineering. Take a concrete target: 10 million transactions per day, a p99 latency budget of 400 ms, and an average transaction that touches 4 ledger accounts.

Start with throughput. Ten million per day averages roughly 120 transactions per second. Real traffic is spiky — Black Friday, payroll runs, subscription-renewal batches — so plan for a 10x peak, about 1,200 TPS. Because each transaction writes into 4 accounts, the ledger sees 4x write amplification: at peak that is close to 5,000 journal rows per second.

Storage compounds fast. A journal row of about 200 bytes across 4 rows per transaction is 10M × 4 × 200 bytes ≈ 8 GB per day of raw journal. That is roughly 3 TB per year, and once you add secondary indexes plus write-ahead-log overhead (budget 2–3x), a realistic figure is 8–10 TB per year for the ledger alone.

These numbers set the reconciliation strategy. If a recon job must sum roughly 200 million rows per day, scanning the full table hourly is a non-starter. Partition the journal by day and maintain materialized per-account running balances so reconciliation reads only the delta since the last checkpoint — O(changes) instead of O(total). And they set the scaling threshold: a single well-tuned Postgres primary can absorb ~1,200 TPS with batched writes; you only shard the ledger by merchant_id once sustained writes push past ~5,000 TPS.

The compliance boundary comes first#

Before architecture, decide what touches raw card data, because that decision draws the most expensive line in the system: PCI-DSS scope. Anything that stores, processes, or transmits cardholder data — the PAN, the magnetic track, the CVV — is in scope. That includes the obvious card-handling services but also logs, backups, and any load balancer terminating TLS on that traffic.

The architectural move that shrinks scope is tokenization. The browser sends the raw card directly to the PSP through a hosted iframe or SDK (Stripe Elements, Adyen's fields), and the PSP returns an opaque token. Your platform stores the token, never the PAN. Now only the iframe's origin is in scope; the hundreds of microservices behind it handle tokens and stay out of audit reach. Layer on network segmentation for the cardholder-data environment, strict egress rules, ephemeral keys from a KMS, and a hard rule that PANs never hit logs, and the compliance surface collapses to a manageable sliver.

The high-level architecture#

The synchronous path stays short. A stateless payment API authenticates the merchant, validates the request, enforces idempotency, and writes authoritative state — then returns. Everything that can wait moves to durable queues.

Fraud checks, PSP calls, ledger posting, payout batching, and notifications all run asynchronously behind the orchestrator and message bus. The caller gets a fast acknowledgment with a payment status; the money movement completes over the following seconds through the saga.

API design and idempotency#

The single most important API contract in payments is idempotency, because networks retry and clients crash mid-request. The client generates a UUID per logical payment attempt and sends it as an Idempotency-Key header.

Server-side, store (idempotency_key, request_hash, response) in a dedicated table with a unique index on the key, scoped by (merchant_id, key). On a retry the server compares the incoming request fingerprint to the stored request_hash: if it matches, return the cached response verbatim; if it mismatches, return 422 because the same key is being reused for a different request — a bug or an attack.

The subtle part is where this runs. Enforcement belongs at the earliest stateful layer — the payment service, not the API gateway — and the key insertion must be transactional with the money-mutating write. If recording the key and committing the charge are separate steps, a crash between them produces exactly the phantom success you were trying to prevent.

The double-entry ledger#

Payments do not store a single "balance" column and mutate it. They use a double-entry ledger, the same model banks have used for centuries. Every movement of money is recorded as two or more journal entries whose signed amounts sum to zero within a transaction: debit one account, credit another.

The core invariant SUM(amount) OVER txn_id = 0 is checked at write time, and a periodic reconciliation asserts that the sum of journal entries per account matches the materialized balance. This model earns its complexity three ways: every cent has auditable provenance, multi-account transfers commit atomically, and bugs surface as a non-zero journal sum you can alert on rather than as silent balance drift you discover from an angry merchant. The journal is append-only — you never edit an entry; a refund posts a new reversing entry.

Coordinating the flow across failures#

A single charge is a distributed transaction across authorization, fraud check, capture, and ledger posting — spread over services that can each fail independently. Distributed XA locks would be a nightmare at 1,200 TPS, so use the saga pattern: model each step as a local transaction with a compensating action.

A saga orchestrator — a state-machine service — durably records the current step in a sagas table and drives the next by emitting commands to Kafka. On failure it walks the compensations in reverse. It is idempotent per (saga_id, step), retries transient errors, and escalates terminal failures to a dead-letter queue for a human. That gives eventual consistency without a two-phase-commit coordinator holding locks across services.

Webhooks need the same discipline in reverse. PSPs deliver at-least-once and out of order. Verify the HMAC signature, upsert into a webhook_events table keyed by provider_event_id to dedupe retries, then process asynchronously through a monotonic state machine that rejects stale transitions — never apply pending after succeeded. Rank statuses and apply last-write-wins by the provider's created_at; reconcile critical events like refund.created by fetching the authoritative object from the PSP API.

Retries to a flaky gateway are safe only because every PSP call already carries your idempotency key. Add exponential backoff with jitter (250ms × 2^n ± 50%, capped at 30s) and a circuit breaker per PSP per card network. Retry only transient errors — timeouts, 5xx, 429 — never terminal 4xx like insufficient_funds. And push long retries into a durable job queue so a pod restart never loses one.

How this shows up in interviews#

Interviewers use payments to test whether you respect correctness under failure. The reliable rhythm: state functional requirements, split auth from capture early, quote the capacity numbers above, then draw the short synchronous path with async orchestration behind it. From there the follow-ups are predictable — "what happens if the client retries the charge?" (idempotency key, transactional with the write), "how do you avoid double charges to the gateway?" (idempotency plus durable retry queue), "how do you know the books are correct?" (double-entry invariant plus reconciliation), and "the ledger primary is saturated, now what?" (partition by day, shard by merchant_id). Name the source of truth, then explain which trade-off you would reverse if the workload shifted.

Further learning#

• Design a Payment System: System Design Interview — High-Performance Programming's walkthrough of the same flow end to end.
• Practice the interactive version: Practice this topic on YeetCode.
• Related designs that reuse these patterns: Design a Search Engine, Design Top K Problem, Design a Proximity Service, and Design YouTube.

FAQ#

Why do payment systems use a double-entry ledger instead of a balance column?#

A single mutable balance column hides errors: if a bug adds money that came from nowhere, the number still looks plausible and you find out later from a mismatched bank statement. Double-entry records every movement as offsetting debit and credit entries that sum to zero per transaction, so any bug shows up immediately as a journal that does not balance. It also gives full auditable provenance — every cent traces to a source account — and lets multi-account transfers commit atomically, which is exactly what regulators and merchant reconciliation demand.

Where should idempotency keys be enforced in a payment API?#

At the earliest stateful layer that owns the money-mutating write — the payment service itself, not the API gateway or a proxy. The key insert must sit inside the same database transaction as the charge, so there is no window where the key is recorded but the charge is not (or vice versa). Enforcing it at the gateway leaves that gap open, and a crash in the gap produces a phantom success or a silent double charge. Store the key with a request fingerprint so a mismatched reuse returns 422 instead of quietly overwriting.

How do you prevent a flaky payment gateway from causing double charges on retry?#

Every call to the gateway carries your platform's idempotency key, so a retry that reaches the PSP after a lost response is deduplicated by the PSP itself. On your side, classify errors: retry only transient failures (network timeouts, 5xx, 429) and never terminal 4xx responses like insufficient funds. Use exponential backoff with jitter and a per-provider circuit breaker to fast-fail during outages, and move long retries into a durable job queue so a process restart cannot lose an in-flight attempt or hold a request thread for minutes.

What is the difference between authorization and capture?#

Authorization asks the card issuer to place a hold on the cardholder's available balance — funds are reserved but no money actually moves. Capture is the second step that pulls those reserved funds and begins settlement to the merchant. They are separate so a business can authorize at checkout and capture only when it ships the goods, run fraud review in between, or split one authorization into several partial captures. Card networks also charge lower interchange when capture happens close to fulfillment, so the two-phase model saves money as well as adding flexibility.

How do you handle out-of-order webhooks from Stripe or Adyen?#

PSPs guarantee at-least-once delivery, not ordering, so a pending event can arrive after a succeeded one. Verify the HMAC signature first, then upsert into a webhook_events table keyed by the provider's event id to discard duplicate retries. Process events asynchronously through a monotonic state machine that ranks statuses and refuses to move a payment backward — applying last-write-wins based on the provider's timestamp. For high-stakes transitions like refunds, periodically fetch the authoritative object straight from the PSP API rather than trusting event order.