system-design Coursesystem-designpaymentsfintechcase-studyappliedadvancedintermediate

Design a Payment System

•11 min read

Design a Payment System

A merchant called us: checkout showed "payment approved," the customer got a confirmation email, but the money never landed in their account. Support pulled the transaction — authorized: yes, captured: no, settled: pending. The customer's card had a hold; our ledger had a gap; the payment service provider (PSP) daily settlement file did not match our internal totals.

I knew idempotency and async processing in the abstract. I had not internalized the invariant that makes payments bite: authorization and settlement are different phases. Approval reserves funds; settlement moves money. Skip that split in the design and support ends up re-explaining card holds while finance reconciles against a PSP file that does not match your ledger.

Below is how I whiteboard a card platform: actors, requirements, napkin math, authorize vs capture APIs, wallet vs ledger, daily reconciliation, and what happens when the PSP times out while the client retries.

Intended audience: developers past the fundamentals who want one composed fintech case — interview whiteboard or an internal design review, not a glossary dump.

Prerequisites:


The Domain Invariant: Authorization ≠ Settlement

Card payments are a two-phase process:

PhaseWhat happensSpeedMoney moves?
AuthorizationIssuer validates card, checks limit/fraud, returns auth codeSecondsNo — amount is reserved
Capture / settlementCapture confirms the charge; settlement moves funds to the merchant (often batched daily)Hours to daysYes

That split drives everything else:

  • You need authorize and capture as separate API steps (or explicit state machine)
  • A txn can be authorized but not captured — fulfillment vs payout decisions
  • Idempotency matters on both steps — timeouts during authorize are ambiguous (idempotency)
  • Reconciliation compares your ledger to the PSP's settlement file — auth success does not guarantee settlement success

If you remember one thing for interviews: approval ≠ paid.


Actors and System Boundary

External entities (outside our system)

EntityRole
CustomerCardholder purchasing goods/services
MerchantSeller; online store collects payment details
Issuer bankCustomer's bank; validates card and credit limit
Acquiring bankMerchant's bank
Payment service provider (PSP)Third-party gateway (Stripe, Adyen, etc.) — routes to card networks; carries most PCI scope
Card networkVisa, Mastercard — network rules and validation

We do not build banks or card networks. We orchestrate calls to the PSP and keep our own wallet, ledger, and reconciliation state.

In scope (our system)

ComponentRole
Payment serviceOrchestrates payment flow, persists events
Fraud / risk checkReal-time scoring before forwarding to PSP
WalletMutable merchant balance (operational)
LedgerImmutable append-only audit trail
ReconciliationMatch ledger vs PSP settlement files
Dispute serviceChargebacks, compliance records

Loading diagram…


Requirements

Functional

RequirementNotes
User registration and authenticationMerchant-facing; hash passwords, session management
Payment processingCredit/debit via PSP; authorize then capture
Transaction historyBy customer and date range
Balance managementMerchant available + pending balances
Mobile accessibilityResponsive merchant UI

Non-functional

RequirementTarget
SecurityPayment Card Industry (PCI DSS); encrypt card data in transit; never log primary account number (PAN) or card verification value (CVV)
AvailabilityHighly available payment path; no single point of failure on orchestration
ReliabilityRedundant services, multi-datacenter deployment, failover
ScalabilityHandle peak sale events; horizontal scale orchestration tier
PerformanceLow-latency authorize path; async settlement acceptable

Out of scope (say this in interviews)

  • Building our own card network or becoming a licensed bank
  • Cryptocurrency rails
  • Full accounting / tax reporting (export to finance systems only)
  • In-person point-of-sale (POS) hardware

Assumptions and Back-of-the-Envelope Math

Assumptions (state these before calculating):

  • 50M transactions per day
  • ~100 bytes per transaction record; ~200 bytes per user record (stored once)
  • Peak ≈ 2.5× average → ~1,400 TPS at peak
  • ~130 bytes average response (status, tokens, metadata)

Storage (per day):

50M × 100 B ≈ 5 GB/day in transaction events (plus indexes, ledger entries)

User records are amortized — transactions reference user_id.

Bandwidth at peak (~1,400 TPS):

Incoming: 1400 × 100 B ≈ 1.1 Mbps
Outgoing: 1400 × 130 B ≈ 1.5 Mbps

Real deployments need headroom for retries, webhooks, reconciliation files, and metadata — multiply by 2–5× for planning.

Servers (illustrative):

If one app server handles ~64K RPS for lightweight health checks but payment orchestration is heavier, do not use health-check RPS for payment pods. Payment path is I/O-bound (PSP latency dominates). Start with horizontal scale behind a load balancer; size from p99 PSP round-trip, not CPU alone.

When math changes the design:

  • 5 GB/day → time-partitioned txn tables or cold archive; ledger never deleted
  • Peak TPS → async capture pipeline acceptable; authorize path stays sync
  • Global merchants → multi-region reads; writes sticky or single primary (multi-region)

High-Level Design

  1. Customer clicks Pay on merchant site → payment service receives event
  2. Payment service → fraud/risk check (decoupled service)
  3. On pass → PSP → issuer bank authorization
  4. Auth code returned → merchant can fulfill (state: AUTHORIZED)
  5. Later: capture job or API → PSP captures funds → wallet updated
  6. Ledger append for every state transition
  7. End of day: reconciliation vs PSP settlement file

Fraud sits in its own service so weekly rule changes are not tied to PSP release windows, and a bad fraud deploy does not take down capture.


API Surface

Interviewers expect a two-step payment API because of authorization vs capture:

APIPurpose
registerUser(username, email, password)Hash password, store user
authenticateUser(username, password)Session / token
authorizePayment(amount, card, merchant_id, idempotency_key)Reserve funds; return authorization_code
capturePayment(authorization_code, amount, idempotency_key)Move funds to merchant
checkPaymentStatus(transaction_id)Poll for async states
getTransactionHistory(customer_id, start, end)History

Pass idempotency_key on authorize and capture. The client generates one key per checkout intent; the server stores key → result and returns the same response on retry. Storage patterns and TTL: Idempotency and Delivery Guarantees.

// Client: one key per checkout intent
const idempotencyKey = `checkout-${cartId}-${userId}`;

await fetch('/payments/authorize', {
  method: 'POST',
  headers: { 'Idempotency-Key': idempotencyKey },
  body: JSON.stringify({ amount, cardToken, merchantId }),
});

Wallet vs Ledger

WalletLedger
PurposeOperational balance — "how much can merchant withdraw?"Audit trail — "what happened, when, between whom?"
MutabilityUpdated in place on capture/refundAppend-only; no updates or deletes
Source of truth forPayout decisionsDisputes, finance, reconciliation
AnalogyChecking account balanceBank statement line items

Flow: capture succeeds → update wallet → append ledger entry (txn_id, amount, parties, status, timestamp).

If wallet and ledger disagree, ledger wins for audit; wallet is rebuilt from ledger in recovery. Event sourcing thinking helps here — even if you do not run full event sourcing, treat the ledger as the immutable log.


Reconciliation

Every day the PSP sends a settlement file summarizing captured transactions and fees.

Reconciliation job:

  1. Load internal ledger entries for status = CAPTURED for the settlement window
  2. Compare totals and line items to PSP file
  3. Flag mismatches (missing capture, amount drift, duplicate settlement)

Example mismatch: merchant ledger shows three payments totaling $225; PSP file shows $200 — investigate before payout.

Reconciliation is not optional for production fintech. Auth success without matching settlement is the incident that started this post.


Fraud and Risk (Decoupled)

Fraud detection — pattern analysis (velocity, device, geo)
Risk check — score → proceed / challenge (3-D Secure, 3DS) / block

Why separate from PSP?

  • Different release cadence (rules change weekly)
  • Different blast radius (bad deploy shouldn't take down PSP path)
  • Scale independently — Black Friday traffic spikes fraud checks before PSP limits

Fallback trade-off: if fraud service times out on a $2 charge, some teams fail open (accept risk); on $2,000, fail closed. Document the policy — interviewers ask.


Reliability Under Retries and Timeouts

I point to earlier posts for the mechanics here instead of re-teaching Kafka in the interview.

FailurePatternWhere to read more
Lost payment eventsPub-sub (Kafka); ack after wallet + ledger writeAsync processing
Transient PSP errorsRetry + exponential backoff + max attemptsReliability
Client timeout on authorizeIdempotency key — ambiguous if charge went throughIdempotency
Double-click PaySame idempotency key on authorizeIdempotency
Fraud service downExplicit fail-open vs fail-closed policyFraud section above

Loading diagram…


How the Design Meets Requirements

Before I stop drawing boxes, I map each requirement row to something buildable — not a label on a diagram.

RequirementTechnique
User authMerchant app + payment service validates session
Payment processingPayment service + PSP + fraud; authorize → capture
Transaction historyLedger query by customer_id + date range
BalanceWallet updated on capture; pending = authorized-not-captured
MobileResponsive merchant UI
SecurityPCI scope via PSP tokenization; no PAN in our DB
AvailabilityRedundant payment/recon services; multi-datacenter; load balancing
ScalabilityAsync capture; horizontal payment pods; DB partitioning by merchant_id
PerformanceSync authorize (user-facing); batch settlement (background)

What Breaks When…

  1. Authorization succeeds, settlement fails — customer sees hold; merchant not paid. Mitigation: txn status machine, reconciliation alerts, automated capture retry with idempotent capture keys.

  2. Timeout after authorize — client retries; without idempotency, double auth hold or duplicate capture. Mitigation: idempotency store with 24h TTL minimum.

  3. Ledger–PSP mismatch at reconciliation — payout blocked until ops resolves. Mitigation: daily recon job, alerting on any delta > $0, runbook for partial PSP outages.


Common Mistakes I Made

  1. Treating authorize and capture as one "charge" API — breaks retries and state tracking
  2. Updating wallet without ledger append — can't explain balance to finance after a bug
  3. Skipping reconciliation until finance finds a hole — build day-one batch job
  4. Logging card numbers during debugging — PCI scope violation; use PSP tokens only
  5. Re-explaining Kafka in the interview instead of naming one sentence and moving to wallet/ledger

Key Takeaways

  1. Authorization ≠ settlement — design APIs and state machines around two phases
  2. Wallet is operational; ledger is audit — append-only ledger for every money movement
  3. Idempotency keys on authorize and capture — timeouts are normal; duplicate charges are not
  4. Reconciliation is production-critical — compare ledger to PSP settlement daily
  5. Decouple fraud from PSP — scale and deploy independently; document fail-open policy
  6. Compose fundamentals, don't re-lecture — load balancing, async, idempotency, and observability show up here by reference

Test Your Understanding

🧩 Initializing quiz...
Quiz ID: system-design-payment-system

Next in series: Fan-Out on Write vs Fan-Out on Read

Happy coding!

Written by Sandeep Reddy Alalla

Share your thoughts and feedback!