Design a distributed message queue

Should a message queue be a log or a broker that deletes on acknowledgement?

When the prompt asks for replay and for a million messages a second, a log. A log appends every message to the end of a file and lets each consumer keep its own position, so replaying yesterday is moving that position back, and a second consumer group costs no extra writes. A broker that tracks and deletes each message gives you per-message acknowledgement, retries and dead letters, but it keeps state for every message and cannot replay what it has deleted. Say which requirement decides it: replay and throughput pick the log.

What does it mean for a published message to be acknowledged?

It means every replica in the in-sync set has appended it, and there are at least two of them. The leader tracks how far each follower has fetched, and the smallest of those positions is the high-water mark: consumers may read below it and not beyond it. If the in-sync set shrinks below two, the leader refuses writes that ask for full acknowledgement rather than acknowledge a message that sits on one disk. That refusal is the design choosing durability over availability, and you should say it in those words.

How many partitions and brokers does a million messages a second need?

Compute both and take the larger. For brokers, a million 1 KB messages a second is 1 GB a second, 3 GB with three copies, which ten brokers can write; but seven days of it is 1.81 PB, and at 40 TB a broker that is 46 brokers. Retention sizes the fleet. For partitions, one consumer handles about 2,000 messages a second, so a group needs at least 500 partitions to keep up, and I would create 1,000 because adding partitions later moves keys between them.

What happens when the broker leading a partition dies?

The coordination service notices the missed heartbeats, picks a new leader from the in-sync replicas, and raises the partition's leader epoch so the old leader cannot write if it returns. With 2 seconds of missed heartbeats and an election, the partition takes no writes for about 3 seconds, and producers hold those messages in their buffers and resend them with the same sequence numbers. At four nines a partition can afford about a thousand of those failovers a year.

How do you handle one key that takes most of the traffic?

A key always lands on one partition, and a partition is read by one consumer in each group, so a dominant key leaves that consumer behind while the others idle. First check that the key is the right one: key by the entity whose order matters, not by a whole tenant. If one entity really dominates, split its key into a few sub-keys so its messages spread across partitions, and accept order within each sub-key. Adding partitions does not help one key.