Idempotency, retries and exactly-once
A request can arrive twice and the effect must happen once. Give every request a key, claim the key before the work runs, and a repeat gets the stored reply instead of a second charge.
What does idempotent mean in an API?
An operation is idempotent when doing it twice leaves the same state as doing it once. Setting a field to a value is naturally idempotent; adding money to a balance is not. For the second kind the server makes it idempotent by recognising a repeat: the client sends a key with the request, the server records the key and its outcome, and a later request with the same key gets the recorded outcome instead of running the work again.
Is exactly-once delivery possible?
Not over a network that can lose a message or its acknowledgement, because the sender cannot tell a lost message from a lost reply and has to resend. What you can build is an exactly-once effect: deliver at least once, and make the receiver skip what it has already applied. That needs the record of what was applied and the effect itself to change in one transaction.
Where should the idempotency key be generated?
On the client, once per logical operation, before the first attempt, and reused on every retry of that operation. A key made fresh on each attempt dedupes nothing. A key made on the server cannot help either, because the retry is a new request the server has never linked to the first. A random UUID is the usual choice.
How long should the server keep idempotency keys?
At least as long as a client may retry, and a margin beyond it. If clients give up after a few minutes, a day is generous. The key rows are small, so the arithmetic is keys per day times row size times days kept; for 21.6 million payments a day at 300 bytes a row and one day, that is 6.48 GB. Expire them with a TTL on the row or a sweep by creation time.
What is a retry storm?
It is what happens when clients retry every timeout at once while the server is already behind. Each retry adds load exactly when there is none to spare, the queue grows, more requests time out, and the retries multiply. The server can stay overloaded after the original cause is gone. Exponential backoff with jitter spreads the retries out, and a retry budget caps them at a fixed share of normal traffic.