Design stream processing with exactly-once
Half a million events a second, counted per user per minute, and not one event counted twice after a crash. The round turns on one sentence: replay from a checkpoint, and make the sink forgive the replay.
The round this design follows
What does exactly-once actually mean in stream processing?
It means the stored result is the same as if no failure had happened, not that each event is touched once. A processor that crashes reads some events twice. Three pieces make the result exact anyway: a source you can rewind to an offset, a checkpoint that saves the state and the offsets together, and a sink where writing the same row again changes nothing. Say it that way in the room, because an interviewer who hears "each event is processed once" will ask what happens on a crash.
How big is the state, and does it decide the design?
On this prompt it does not. Ten million keys, each holding the current window and the one before it while late events may still arrive, at 100 bytes a window, is 2 GB. Spread over 20 nodes that is 100 MB each. What decides the design is recovery: how much a crash replays, and how fast the task catches up.
What is a watermark?
It is the processor's estimate of how far event time has got: a claim that no event older than the watermark is still on its way. A window emits its result when the watermark passes its end. An event that arrives after that is late; within the 30-second allowance it updates the window and the row is written again with a higher version, and after it the event goes to a separate log of late events.
Why not use a transactional sink instead of idempotent upserts?
A transactional sink commits the output of each checkpoint in one transaction, which works and is what some processors offer. It delays every result until the checkpoint completes, up to 10 seconds here, and it needs a sink that supports those transactions. An upsert keyed by user and window with a version needs neither: a replay rewrites the same rows, and the version rejects an older row that arrives after a newer one.
What happens when one user sends most of the events?
That user's task takes most of its partitions' load and falls behind while the others idle. Split the hot key into a few sub-keys with a salt taken from the event id, count each sub-key on a different task, and add the partial counts when the window closes. The salt must come from the event, not from a random number, so a replay sends each event to the same sub-key as before.