Design a notification system
Ten million notifications on an ordinary day, a hundred million on a campaign day, and three providers that each take a thousand a second. The round turns on one sentence: the provider sets the pace, so a queue per channel takes the load and a dedupe key makes the retries safe.
The round this design follows
Why put a queue between the producers and the providers?
Because the provider, not your servers, sets the pace. Each channel's provider takes about a thousand sends a second, and a campaign day averages 1.16 thousand a second, more than one provider takes. A producer that calls the provider inline makes its own users wait on the provider's latency and fails when the provider is slow. A queue per channel lets the producer hand off in the time of one database write and lets workers drain at the rate the provider accepts.
Can a notification system deliver exactly once?
Not end to end, and it is worth saying so plainly. The queue delivers at least once, because a worker can die after the provider accepted a send and before it recorded that. A dedupe key per event, user and channel, kept for 24 hours, turns almost every redelivery into a skip. What remains is the rare send that landed while its worker died, which goes out twice; the honest answer is at least once plus dedupe, and it is what people mean by exactly once in practice.
How does a campaign of a million users not block password resets?
Give each channel two lanes, urgent and bulk, and reserve a share of the provider's cap for the urgent lane. A million campaign messages at a thousand a second take about seventeen minutes to send; in one first-in, first-out lane a password reset would wait behind all of them. With lanes the campaign drains a little slower and the reset goes out in the time the send itself takes.
What is the outbox pattern and why does this design need it?
The service writes the notification row and an outbox row in one database transaction, and a relay publishes the outbox rows to the queue afterwards. Without it, the service writes the row and publishes as two separate steps, and a crash between them leaves a notification that was saved and never sent, or sent and never saved. The outbox makes the write and the publish succeed or fail together, at the cost of a relay process and a short delay before the send.
What happens when a provider slows down or goes down?
The channel's queue grows, and nothing is lost, because the messages are durable in the queue until a worker has a connection to send them on. Workers retry with a delay that doubles each attempt, up to the retry budget of three, and then park the message on a dead-letter queue for a person or a fallback channel. The other channels are unaffected, because each has its own queue and its own workers.