Availability: nines, copies and failure domains

• Both layouts saw the same failures. Only the placement differs.
• Copies help only against failures that hit them one at a time.

Three copies in one zone are as available as the zone. I put one copy in each zone, so a zone failure takes one copy, not all.

Downtime = (1 - target) × period. A month is 30 days; a year is 365.

99.9% is 43 minutes a month. 99.99% is about 4 minutes, which is less than one manual failover takes.

Burn rates for a 99.9% SLO, from the multiwindow, multi-burn-rate recipe in Google's SRE Workbook. Budget lasts = 30 days / rate.

My SLI is the share of good requests at the load balancer. The SLO is 99.9% over 30 days, so the budget is 43 minutes of full outage.

N is the number of copies the peak load needs. Active-active needs each record to have one home region, or a conflict rule.

I run N+1 across three zones for stateless tiers, and a standby database in another zone. A second region comes only when the SLO needs it.

chain(a1 .. an)   = a1 × a2 × ... × an        // up only if every part is up1
copies(a, n)      = 1 - (1 - a)^n             // down only if every copy is down2
at_least(k, n, a)3 = sum for j = k..n of C(n, j) · a^j · (1 - a)^(n - j)

zones:                              // copies in one zone fail together
  total = 0
  FOR EACH set S of zones that are up4:
    p = chance that exactly S is up
    FOR EACH tier: p = p × at_least(need, copies in S, a)
    total = total + p
  availability = region × total × each external dependency5

// 3 copies at 99.5%, 1 zone at 99.9%:   0.999 × (1 - 0.005³)       = 99.9%6
// 3 copies at 99.5%, 1 per zone:        1 - (1 - 0.999 × 0.995)³   = 99.99998%

// Downtime is the time a service may be down in a period and still meet the target.
func Downtime(availability, periodSeconds float64) float64 {
  return (1 - availability) * periodSeconds
}

// Redundant is the availability of n copies when any one copy is enough and copies fail
// independently: the service is down only when every copy is down.
func Redundant(availability float64, n int) float64 {
  return 1 - math.Pow(1-availability, float64(n))
}

// Chain is the availability of components that a request needs one after another: the request
// succeeds only when every component is up.
func Chain(availability float64, n int) float64 {
  return math.Pow(availability, float64(n))
}

// AtLeast is the chance that at least k of n independent copies are up, each with availability a.
// With k = 1 it is the redundancy formula: down only when every copy is down.
func AtLeast(k, n int, a float64) float64 {
  if k <= 0 {
    return 1
  }
  if k > n {
    return 0
  }
  if k == 1 {
    return estimate.Redundant(a, n)
  }
  p := 0.0
  for j := k; j <= n; j++ {
    p += binomial(n, j) * math.Pow(a, float64(j)) * math.Pow(1-a, float64(n-j))
  }
  return p
}

// Series is the availability of parts a request needs one after another.
func Series(as ...float64) float64 {
  p := 1.0
  for _, a := range as {
    p *= a
  }
  return p
}

// inZones counts the instances of a tier that live in the up zones; up is a bit set of zones.
func inZones(t Tier, zones int, up uint) int {
  m := 0
  for i := range t.Instances {
    if up&(1<<(i%zones)) != 0 {
      m++
    }
  }
  return m
}

// zoneLayer is one region's internal tiers, with zone failures. It sums over every set of up
// zones: the chance of that set, times the chance that every tier has enough instances up in it.
// A zone is shared by every tier, so the tiers are independent only once the zones are fixed.
func zoneLayer(tiers []Tier, zones int, zoneAvail float64) float64 {
  total := 0.0
  for up := uint(0); up < 1<<zones; up++ {
    p := 1.0
    for z := range zones {
      if up&(1<<z) != 0 {
        p *= zoneAvail
      } else {
        p *= 1 - zoneAvail
      }
    }
    for _, t := range tiers {
      if !t.External {
        p *= AtLeast(t.Need, inZones(t, zones, up), t.Avail)
      }
    }
    total += p
  }
  return total
}

// regions combines one region's availability r over the region mode.
func regions(a Arch, r float64) float64 {
  switch a.Mode {
  case ActiveActive:
    return estimate.Redundant(r, 2)
  case ActivePassive:
    // While the primary is down, the standby serves if the failover works, once it is done.
    lost := math.Min(1, a.RTOMin/a.OutageMin)
    return r + (1-r)*a.FailoverOK*r*(1-lost)
  default:
    return r
  }
}

// availability is the whole deployment: regions, then the external dependencies in series.
func availability(a Arch) float64 {
  r := a.RegionAvail * zoneLayer(a.Tiers, a.Zones, a.ZoneAvail)
  p := regions(a, r)
  for _, t := range a.Tiers {
    if t.External {
      p *= t.Avail
    }
  }
  return p
}

A request path multiplies. Copies multiply their failures. With zones, I add up over which zones are up, because a zone is shared by every tier.

I name the failure domain each copy protects against. Copies on one host protect against a crash, not against a zone.

Multi-zone gives about 99.9864%; the region is then the weakest part. A second region, active-active, gives 99.99999815%.

each part: up for Exp(MTBF) hours, then down for Exp(MTTR) hours
take the earliest change from a queue; flip that part
service up = every tier has enough copies up in up zones
measured = 1 - down hours / all hours

// Simulate runs l for `years` and measures the share of time every tier had enough instances up
// in zones that were up. Each part alternates between up and down; the next change is always the
// earliest one in a queue. With trace set, it also returns the down intervals of every part and of
// the service in the first traceHours.
func Simulate(l Layout, years float64, seed uint64, traceHours float64) (SimResult, map[string][]Interval) {
  ps := parts(l, seed)
  q := make(queue, len(ps))
  copy(q, ps)
  heap.Init(&q)

  end := years * 8760
  trace := map[string][]Interval{}
  mark := func(id string, from, to float64) {
    if from < traceHours {
      trace[id] = append(trace[id], Interval{from, min(to, traceHours)})
    }
  }
  downSince := map[*part]float64{}

  res := SimResult{Years: years}
  counts := make([]int, len(l.Tiers))
  now, up, serviceDownAt := 0.0, true, 0.0
  for q[0].next < end {
    p := q[0]
    now = p.next
    res.Events++
    p.up = !p.up
    if p.up {
      mark(p.id, downSince[p], now)
      p.next = now + p.rng.ExpFloat64()*p.fail.MTBF
    } else {
      downSince[p] = now
      p.next = now + p.rng.ExpFloat64()*p.fail.MTTR
    }
    heap.Fix(&q, 0)

    nowUp := serviceUp(l, ps, counts)
    switch {
    case up && !nowUp:
      serviceDownAt = now
      res.Outages++
    case !up && nowUp:
      res.DownHours += now - serviceDownAt
      mark("service", serviceDownAt, now)
    }
    up = nowUp
  }
  if !up {
    res.DownHours += end - serviceDownAt
  }
  res.Measured = 1 - res.DownHours/end
  return res, trace
}

I checked the formula against a simulation of failures. It agrees within 10%, and the naive formula is off by hundreds of times when copies share a zone.

A zone failure is handled inside the region in seconds. A region failure needs detect, promote, fence and shift traffic, and I practise it.

Postgres gives me replicas, promotion and point-in-time recovery. Health checks and DNS move traffic. The service hides short failovers with retries.

• A replica copies a bad DELETE at once. Only a backup brings the rows back.
• Restore a backup on a schedule, into a scratch database, and check it.
• Time the restore. That time is your RTO for the backup path.

RPO is how much data I can lose; RTO is how long recovery takes. A backup I have never restored gives me neither number.

1 TB at 8 MB/s: 1,048,576 MB / 8 = 131,072 s, about 36 h. Parallel restore and physical backups go faster; measure yours.

My restore ran at 8 MB a second, so 1 TB would take about 36 hours on one process. That is the RTO if the replicas are also bad.

Upper bound: SLO ≤ product of the critical dependencies. Two at 99.95% allow at most 99.9%.

With the payment provider at 99.95% in series, multi-zone checkout drops to 99.9364%. I take the provider off the critical path.

• A cell is a complete stack for a fixed set of users or tenants.
• A thin router maps each user to a cell. Keep it simple and highly available.
• Cells do not raise the average much. They cap the worst incident.

I split users into cells, each a full copy of the stack. A bad deploy or a poison request reaches one cell first.

Each copy at 99.5%, each zone 99.9%, each region 99.99%: assumptions, so read the steps as orders of magnitude. Each step costs more to run and to test.

I start with two copies of each tier across zones. A second region comes only when the SLO needs more than one region can give, or a regulator asks.