Design a payment system with a ledger
Twenty million payments a day, a card provider you do not own, and money that must be charged exactly once and always add up. The round turns on two things: an idempotency key that makes every retry safe, and a ledger of paired entries that is only ever appended to.
The round this design follows
How do you make sure a card is charged exactly once?
The client sends an idempotency key with the charge, and the server inserts it under a unique index before it calls the provider, so only one attempt can hold it. The same key is passed on to the provider, which dedupes on its side too. A retry with the same key gets the stored reply, or asks the provider again when the first outcome is unknown, and the provider returns the charge it already made. Exactly once is at-least-once delivery made safe by a key on both sides of the call.
Why a double-entry ledger instead of a balance column?
Every payment writes two entries of equal amount, a debit and a credit, so the sum of all entries is always zero and any error shows up as an imbalance. Entries are only appended, never updated, which gives an audit trail kept forever and removes the row lock a balance column takes on every payment. The balance becomes a derived number, summed from entries and cached. The cost is a rollup job and a balance that can lag the entries by a moment.
What happens when the payment provider times out?
After 10 seconds without an answer the payment is recorded as unknown, never as failed, because the charge may have landed. A retry with the same key asks the provider again, and the provider answers with the charge it made or makes it once. A resolver job does the same for unknown payments no client retried, and the nightly reconciliation against the provider settlement file catches anything left.
Why use an outbox for payment events?
The order service and the receipt must hear about a payment exactly when the ledger records it. Writing the ledger and then publishing to a queue is two systems, and a crash between them loses the event or sends one for a payment that rolled back. The outbox row is written in the same transaction as the entries, and a relay publishes it afterwards, so the event exists if and only if the payment does. Consumers still dedupe by payment id, because the relay delivers at least once.
How do you size a payment system at a thousand payments a second?
A thousand payments a second at about 500 ms a provider call is 500 calls in flight, and 10 thousand if the provider slows to its 10-second timeout, so the connection pool to the provider is the first thing to size. The ledger is two thousand entries a second, which one relational primary handles. Storage decides the shard count: about 6.57 TB a year, kept forever.