Failure and recovery

The last ten minutes are where you kill your own boxes. For each one, say how it breaks, the metric that tells you, the mechanism that heals it, and what the healing costs in data. Then close in thirty seconds.

How do I talk about failure modes in a system design interview?

Box by box, starting with the one that holds the data the requirements care about most. For each, name a concrete failure (it dies, it slows, it falls behind, it fills), the metric that fires when it happens, and the mechanism that heals it. Then say what the recovery costs: writes lost in a window, or a duplicate a retry could create. Two or three boxes done that way beat seven boxes named in passing.

Why is "we would add retries" a weak answer?

It names no failure, no signal and no mechanism, so there is nothing for the interviewer to grade. Worse, a bare retry makes an outage longer: every client retries at the same moment the box comes back, and a retried write that had in fact succeeded is applied twice. The strong form says retries with exponential backoff and jitter, capped, on an idempotent request, behind a circuit breaker.

What is the difference between detection time and recovery time?

Detection is how long it takes the system to decide a box is gone, usually a number of missed health checks. Recovery is how long the fix takes once it starts, such as promoting a replica. Users wait for both, so the outage window is their sum, and the work that piles up is the arrival rate times that window. Shortening detection is often the cheaper half.

Does failover lose data?

It depends on how the replica was kept up to date. With asynchronous replication, the writes the primary acknowledged but had not yet shipped are lost when a replica is promoted, so the loss is the write rate times the replication lag. With synchronous replication to at least one replica, nothing acknowledged is lost, at the cost of a slower write. Say which one your design uses and what it loses.

How much time should failure and recovery get in a sixty-minute round?

Minutes 50 to 58, then a thirty-second close. If you are behind at minute 50, drop whatever you are drawing mid-sentence and move, because this phase carries the third line of the rubric and most candidates never reach it. With only three minutes, take the one box where the data lives, give its detect-then-recover pair, and close.