Observability and capacity
Count requests into latency buckets instead of keeping a line for each one. The p99, the error budget and the capacity plan all come from a few counters whose size never grows with the traffic.
Why use a histogram for latency instead of an average?
An average hides the slow requests people complain about: a thousand requests at 40 ms and ten at 4 seconds average about 80 ms, which looks healthy. A histogram keeps a count per latency bucket, so you can read any percentile, such as the p99 (the latency that 99% of requests beat). It costs one counter per bucket, the same memory at ten requests a second or ten thousand. Keep a running sum beside it and you still have the mean for capacity planning.
How accurate is a p99 read from histogram buckets?
It is exact to the bucket, not to the millisecond. The read finds the bucket that holds the 99th-percentile request and reports its upper edge, so the answer is "at or below 500 ms", never 431 ms. That is why bucket edges are a design decision: put one exactly on your SLO threshold, and pack them densest where you make decisions. Interpolating inside the bucket gives a smoother estimate, but it is still an estimate.
What is an error budget burn rate?
An SLO such as "99% of requests under 250 ms over 30 days" allows 1% of requests to be slow; that 1% is the error budget. The burn rate is the share of requests that were slow in some window divided by the share the SLO allows. A burn rate of 1 spends the budget in exactly 30 days. A burn rate of 14.4 spends 2% of it in an hour, and would empty it in about two days.
Why alert on two windows instead of one?
A long window alone, such as an hour, is slow to fire and keeps paging for up to an hour after the fix, because the bad minutes take that long to leave it. A short window alone, such as five minutes, pages on a one-minute blip that barely touches the budget. Paging only when both burn above the threshold ignores the blip and stops within minutes of the fix. The long window says the loss is real; the short one says it is still happening.
How does Little's law help with capacity planning?
Little's law says the number of requests inside a system equals the arrival rate times the mean time each one spends there. At 3,000 requests a second and a mean of 40 ms, 120 requests are in flight at any moment, so you need at least 120 workers. Divide by a target utilisation such as 70% to leave room to drain bursts, and you get 172 workers, or 11 instances of 16. It holds for any stable queue, whatever the arrival pattern.