Queues and stream processing
Put the work in a line and answer the caller now. The line absorbs the burst, the workers take it at their own pace, and a message is only forgotten when a worker says it is done.
Problems worked on this page, and more to practise
When should I put a queue between two services?
When the caller does not need the result to answer its own user, when the work arrives in bursts the workers cannot absorb as they come, or when the work has to survive a crash of the service that asked for it. Resizing an uploaded photo, sending an email and updating a search index all qualify. Checking a password or reading a balance the user is looking at does not, because the answer is the reason for the request. If you cannot say what the caller does while the message waits, the queue is in the wrong place.
What is a visibility timeout?
It is how long a message stays hidden after a worker takes it. The queue does not delete the message on delivery; it hides it and waits for an ack. If the ack arrives in time the message is deleted. If the worker crashes and no ack comes, the timeout runs out and the message is handed to another worker. Set it longer than the slowest normal piece of work, or healthy work gets delivered twice.
Why is a queue at-least-once and not exactly-once?
Because the queue cannot tell a worker that crashed from a worker that is slow or whose ack was lost on the way. In all three cases it sees no ack before the deadline, and the only safe move is to deliver again. So the same message can be processed twice. Exactly-once effects come from the consumer: an idempotency key, or a write that records the message id in the same transaction as its effect.
What is the difference between a work queue and a log with consumer groups?
A work queue hands each message to one worker and deletes it on ack, so adding workers splits the work. A log keeps every event in order and lets each consumer group keep its own offset, so billing, email and analytics each read every event at their own pace. Use a work queue for tasks, and a log for events that several systems react to or that you may need to replay.
What is a dead-letter queue for?
It holds messages that failed too many times. Without one, a message that always fails (a malformed payload, a bug for one kind of input) is retried forever and takes a worker every time. With a try limit, the message is moved aside after the last try, the workers carry on with the rest, and someone can read the parked messages, fix the cause, and replay them.