Design a digital wallet
Ten million users, five thousand transfers a second, and not one cent moved twice. The round turns on one decision: balances are derived from a log of events, one writer per shard decides every debit, and a transfer between shards is a saga.
The round this design follows
Should a wallet store balances or a log of events?
A log of events, with balances derived from it. Each transfer appends a debit to one account and a credit to the other, and a balance is the sum of an account's events, kept current in a view that the projector updates. The log is the audit trail the prompt asks for, so there is no second table to keep in step, and any balance at any past moment is a replay. A balance column updated in place is simpler to read, but it forgets how it got there.
How do you make a transfer happen exactly once?
The client picks the transfer id and sends it with every retry, and the id is the idempotency key. The shard that owns the payer's account has one writer, and before it debits it looks the id up among the transfers that account has applied; a repeat gets the first answer and moves nothing. The credit on the other shard carries the same id and is checked the same way, so a saga step retried after a failover lands once.
Why use a saga and not two-phase commit for transfers between shards?
With 20 shards, 95 percent of transfers cross two of them, so the cross-shard path is the main path. Two-phase commit holds a lock on both balances for the coordinator's round trips, and a coordinator that dies between prepare and commit leaves them locked until it comes back. A saga debits the payer with an outbox entry in one append, credits the payee under the same transfer id, and refunds the payer if the credit is refused. For that moment the amount sits on the transfer row, in neither balance, and you say so.
How many shards does a wallet at five thousand transfers a second need?
Count commands, not transfers. Each transfer is a debit and a credit, so the peak is 10 thousand commands a second. One shard's single writer does about a thousand a second with its appends batched, and I keep each shard at half its limit, so that is 20 shards of 500 commands a second each. Storage never decides it: 288 million events a day at 100 bytes is 10.5 TB a year before copies.
What happens when one merchant receives a large share of all transfers?
Its account lives on one shard with one writer, so every credit to it queues there. At one transfer in five, that is a thousand credits a second on top of the shard's own 500, against a limit of a thousand. Batch the merchant's row in the balance view first, and if the writer is still past its limit, split the account into four sub-accounts on four shards: a credit goes to the one its transfer id hashes to, and the balance is their sum.