Consistency models
A consistency model says which values a read may return when there are many copies and many clients. Stronger models look more like one machine, and cost more.
- 1Linearizable: one copy in real time. Sequential: one order, real time ignored. Causal: causes before effects.
- 2Causal consistency equals the four session guarantees together, and it can stay available in a partition.
- 3CAP: during a partition, choose availability or linearizability. PACELC: otherwise, choose latency or consistency.
- 4Across regions, a linearizable operation costs at least one round trip to a majority: tens of ms.
- A write takes time to reach every copy.
- A read on a lagging copy returns an older value.
- Each model draws the line at a different place.
With copies, two users can see the same data in different states. The consistency model says which states are allowed.
Availability in a partition from Bailis et al., Highly Available Transactions, VLDB 2014. Sticky: the client keeps using one copy that has its writes.
Linearizable is the strongest and cannot stay available in a partition. Causal can, if each client keeps using one copy.
| model | stale read | new, then old | two keys | concurrent orders | own write lost | goes back | writes reordered | reply first |
|---|---|---|---|---|---|---|---|---|
| Linearizable | Prevented | Prevented | Prevented | Prevented | Prevented | Prevented | Prevented | Prevented |
| Sequential | Allowed | Allowed | Prevented | Prevented | Prevented | Prevented | Prevented | Prevented |
| Causal | Allowed | Allowed | Allowed | Allowed | Prevented | Prevented | Prevented | Prevented |
| Read your writes | Allowed | Allowed | Allowed | Allowed | Prevented | Allowed | Allowed | Allowed |
| Monotonic reads | Allowed | Allowed | Allowed | Allowed | Allowed | Prevented | Allowed | Allowed |
| Monotonic writes | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Prevented | Allowed |
| Writes follow reads | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Prevented |
| Eventual | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
Each column is one example history in the checker below. Every model refuses a value that no client wrote.
I name the anomaly the product cannot show, then pick the weakest model that prevents it.
| user action | with an eventual read | model it needs | status |
|---|---|---|---|
| Save a profile, then reload | The old profile comes back. | Read your writes | Session |
| Scroll a feed, then refresh | A post appears, then disappears. | Monotonic reads | Session |
| Read a thread | A reply shows without its post. | Causal | Causal |
| Buy the last seat | Two people buy it. | Linearizable | Strong |
| Take a unique username | Two accounts get the same name. | Linearizable | Strong |
| Revoke access, then test it | The old permission still works. | Linearizable | Strong |
| Count likes on a post | The count is a little behind. | Eventual | Strong not needed |
I use strong reads where a stale answer costs money or trust, and session guarantees everywhere a user only needs to see their own actions.
A changes a price. B sees the new price and finishes. C reads after that and sees the old price.
One copy, in real time. Each operation takes effect at one instant between its start and its end.
Red: the smallest set of operations that breaks the model.
The checker searches every order for linearizable and sequential, and follows causal chains for the rest. Values start at 0.
To test a claim, I draw the history. Then I ask if one order of the operations explains every read and keeps the rules of the model.
Step 1: Linearizable write
- Send every write to the leader.
- The leader waits until a majority of copies has the write, then acknowledges.
If it fails
The leader cannot reach a majority: the write blocks or fails. This side of a partition chooses consistency.
Writes go to the leader and wait for a majority. Each read picks its copy by the guarantee it needs: leader for strong, a follower with a token for session, any follower for eventual.
- C is linearizability: every read returns the latest write.
- A: every request to a live copy gets an answer. A slow answer still counts.
- P is a fact of networks, not a choice. The choice is C or A while it lasts.
Gilbert and Lynch, 2002, prove the theorem for a read/write object with atomic (linearizable) consistency.
During a partition, a copy that cannot reach the others must either answer with what it has or refuse. Without a partition, I can have both.
| setting | partition | else |
|---|---|---|
| Cassandra ONE for reads and writes | A | L |
| Cassandra QUORUM, R + W > N | C | C |
| DynamoDB global tables, last writer wins | A | L |
| DynamoDB global tables, multi-Region strong | C | C |
| Postgres one leader, synchronous standby | C | C |
| Postgres reads on asynchronous standbys | A | L |
Abadi, 2012: if there is a partition (P), trade availability (A) and consistency (C); else (E), trade latency (L) and consistency (C). Many stores choose per request.
Most of the time there is no partition, so the real daily trade is latency against consistency.
| tool | capability | what it gives this design | also used for |
|---|---|---|---|
| Postgres | One primary takes every write; reads on it see each commit | Linearizable reads and writes, as long as all traffic goes to the primary. | Counters, unique names |
| Postgres | synchronous_commit = remote_apply | The commit waits until the standby has applied it, so a standby read sees it. | Read-your-writes without tokens |
| Postgres | pg_last_wal_replay_lsn() on a standby | A session token: the standby can tell whether it has applied a given commit. | Monotonic reads across standbys |
| Redis | WAIT numreplicas timeout | Limit The documentation states that WAIT does not make Redis a strongly consistent store. | Fewer lost writes on failover |
| etcd | Linearizable reads by default; serializable reads as an option | Strong reads through consensus, or faster reads that may be stale. | Leader election, configuration |
| ZooKeeper | Updates from a client apply in the order sent; sync() before a read | Per-client order. A read after sync() sees the latest update. | Locks, membership |
| DynamoDB | ConsistentRead = true | The most up-to-date data. Not supported on global secondary indexes. | Read-after-write on one table |
| DynamoDB | Global tables: last writer wins, or multi-Region strong (3 Regions) | Choose per table: local writes that can conflict, or writes copied to another Region first. | Multi-region apps |
| Cassandra | Consistency level per query; lightweight transactions on Paxos | R and W per request. IF NOT EXISTS is linearizable, at a higher cost. | Unique inserts |
| MongoDB | Causally consistent sessions | All four session guarantees, with majority read and write concern. | Read-after-write on secondaries |
| Cosmos DB | Five levels: strong, bounded staleness, session, consistent prefix, eventual | Session is the default: read-your-writes for the client that holds the session token. | Global apps |
| Spanner | External consistency, using TrueTime | Transactions behave as if run one at a time, in real-time order. | Global ledgers |
| CockroachDB | Serializable by default; no stale reads | Close to strict serializability, but not quite: clock skew limits the real-time guarantee. | Distributed SQL |
| S3 | Strong read-after-write for PUT and DELETE (since December 2020) | A read after a successful write returns that write. | Object storage as a source of truth |
I pick the store by the guarantees it documents: a consistent read flag, a consistency level per query, a causal session, or consensus by default.
find_order(history, must_precede): // depth-first search
IF every op is placed: RETURN order // the witness1
FOR EACH op not yet placed:
IF an unplaced op must_precede it: skip
IF op is a read AND reg[op.key] != op.value2: skip
place op; IF op is a write: reg[op.key] = op.value
IF find_order(rest) succeeds: RETURN it
undo op // backtrack
remember (placed set, reg) as a dead end3
RETURN none
linearizable: must_precede(a, b) = a.end < b.start4
sequential: must_precede(a, b) = same client AND a first5- 1The order found is the proof. The checker numbers the operations in this order.
- 2A read can come next only if the register holds the value it returned.
- 3The same ops placed with the same register values fail the same way. Skip them next time.
- 4An operation that ended before another started comes first. Overlapping operations go either way.
- 5Only each client’s own order counts. Real time between clients does not.
Tested source Go: the order search
// findOrder looks for one total order of all ops in which each read returns the latest write on
// its key (0 if none), and op a comes before op b whenever mustPrecede(a, b). It tries every op
// whose predecessors are already placed, applies it to the registers, and backtracks on a read
// that does not match. Dead ends are remembered by (ops placed, register values).
func findOrder(h History, mustPrecede func(a, b Op) bool) ([]int, bool) {
n := len(h.Ops)
placed := make([]bool, n)
order := make([]int, 0, n)
regs := map[string]int{}
dead := map[string]bool{}
var place func() bool
place = func() bool {
if len(order) == n {
return true
}
memo := stateKey(placed, regs)
if dead[memo] {
return false
}
for i, o := range h.Ops {
if placed[i] || !ready(h, placed, o, mustPrecede) {
continue
}
if o.Kind == Read && regs[o.Key] != o.Value {
continue // this read cannot come next: the register holds another value
}
prev, had := regs[o.Key]
if o.Kind == Write {
regs[o.Key] = o.Value
}
placed[i] = true
order = append(order, i)
if place() {
return true
}
order = order[:len(order)-1]
placed[i] = false
if o.Kind == Write {
if had {
regs[o.Key] = prev
} else {
delete(regs, o.Key)
}
}
}
dead[memo] = true
return false
}
if !place() {
return nil, false
}
return order, true
}
// ready reports whether every op that must precede o is already placed.
func ready(h History, placed []bool, o Op, mustPrecede func(a, b Op) bool) bool {
for j, p := range h.Ops {
if !placed[j] && j != o.ID && mustPrecede(p, o) {
return false
}
}
return true
}
- The lab compares this search with a check of every permutation on 400 random histories.
- Each refusal shows the smallest set of operations that has no valid order.
A history is linearizable if one total order explains every read and keeps real-time order. The order itself is the proof.
before = each client's own order
+ write -> each read that returns it1
closed under transitivity2
causal(history):
FOR EACH read r, FOR EACH write w on r.key:
IF w before r AND r returned something older than w:
RETURN refused (w, r) // r missed w3
RETURN allowed
// how the chain reaches r's client names the session guarantee4:
// read your writes: r's client wrote w
// monotonic reads: r's client had read w
// monotonic writes: it read a later write by w's client
// writes follow reads: it read a write made after w was read- 1Reading a value makes the reader depend on the write.
- 2Chains count. A reply to a reply depends on the first post.
- 3The read returned the initial value, or a write that happens before w.
- 4On random histories, the lab finds causal equal to all four guarantees together.
Tested source Go: causal · Go: session guarantees
// Causal checks causal consistency: no read returns a value that its own causal past has
// overwritten. Concurrent writes may appear in any order, and different clients may disagree.
func Causal(h History) Verdict {
if r := h.thinAir(); r >= 0 {
return Verdict{Bad: []int{r}, Rule: RuleThinAir}
}
c := newCausalOrder(h)
for _, a := range h.Ops {
if c.before[a.ID][a.ID] {
return Verdict{Bad: []int{a.ID}, Rule: RuleCycle}
}
}
for _, r := range h.Ops {
if r.Kind != Read {
continue
}
for _, w := range h.Ops {
// A newer write on the same key happens before the read, and the read returned
// something older: the initial value, or a write that happens before w.
if w.Kind != Write || w.Key != r.Key || !c.before[w.ID][r.ID] || !c.older(c.src[r.ID], w.ID) {
continue
}
if c.src[r.ID] < 0 {
return Verdict{Bad: []int{w.ID, r.ID}, Path: c.path(w.ID, r.ID), Rule: RuleInitRead}
}
return Verdict{Bad: []int{c.src[r.ID], w.ID, r.ID}, Path: c.path(w.ID, r.ID), Rule: RuleOverwritten}
}
}
return Verdict{OK: true}
}
// ReadYourWrites: after a client writes a key, its reads of that key never return anything older.
func ReadYourWrites(h History) Verdict {
if r := h.thinAir(); r >= 0 {
return Verdict{Bad: []int{r}, Rule: RuleThinAir}
}
c := newCausalOrder(h)
for _, w := range h.Ops {
for _, r := range h.Ops {
if w.Kind == Write && r.Kind == Read && r.Key == w.Key && programBefore(w, r) && c.older(c.src[r.ID], w.ID) {
return Verdict{Bad: []int{w.ID, r.ID}, Path: []int{w.ID, r.ID}, Rule: RuleRYW}
}
}
}
return Verdict{OK: true}
}
// MonotonicReads: once a client has read a value, its later reads of that key never return
// anything older.
func MonotonicReads(h History) Verdict {
if r := h.thinAir(); r >= 0 {
return Verdict{Bad: []int{r}, Rule: RuleThinAir}
}
c := newCausalOrder(h)
for _, r1 := range h.Ops {
for _, r2 := range h.Ops {
if r1.Kind == Read && r2.Kind == Read && r1.Key == r2.Key && programBefore(r1, r2) &&
c.src[r1.ID] >= 0 && c.older(c.src[r2.ID], c.src[r1.ID]) {
return Verdict{Bad: []int{r1.ID, r2.ID}, Path: []int{c.src[r1.ID], r1.ID, r2.ID}, Rule: RuleMR}
}
}
}
return Verdict{OK: true}
}
// MonotonicWrites: a client that sees a client's second write also sees its first.
func MonotonicWrites(h History) Verdict {
if r := h.thinAir(); r >= 0 {
return Verdict{Bad: []int{r}, Rule: RuleThinAir}
}
c := newCausalOrder(h)
for _, w1 := range h.Ops {
for _, w2 := range h.Ops {
if w1.Kind != Write || w2.Kind != Write || !programBefore(w1, w2) {
continue
}
if v, ok := sawThenMissed(h, c, w2.ID, w1); ok {
return Verdict{Bad: append([]int{w1.ID}, v...), Path: append([]int{w1.ID}, v...), Rule: RuleMW}
}
}
}
return Verdict{OK: true}
}
// WritesFollowReads: a client's write comes after everything the client had read, directly or
// through earlier reads. A client that sees the write also sees those values.
func WritesFollowReads(h History) Verdict {
if r := h.thinAir(); r >= 0 {
return Verdict{Bad: []int{r}, Rule: RuleThinAir}
}
c := newCausalOrder(h)
for _, w1 := range h.Ops {
for _, w2 := range h.Ops {
// w1 is another client's write that w2's writer had seen before writing w2.
if w1.Kind != Write || w2.Kind != Write || w1.Proc == w2.Proc || !c.before[w1.ID][w2.ID] {
continue
}
if v, ok := sawThenMissed(h, c, w2.ID, w1); ok {
p := append(c.path(w1.ID, w2.ID), v[1:]...)
return Verdict{Bad: append([]int{w1.ID}, v...), Path: p, Rule: RuleWFR}
}
}
}
return Verdict{OK: true}
}
// sawThenMissed finds a client that reads write w2 and later reads w1's key and gets something
// older than w1. It returns [w2, that read, the later read].
func sawThenMissed(h History, c causalOrder, w2 int, w1 Op) ([]int, bool) {
for _, r := range h.Ops {
if r.Kind != Read || c.src[r.ID] != w2 {
continue
}
for _, r2 := range h.Ops {
if r2.Kind == Read && r2.Key == w1.Key && programBefore(r, r2) && c.older(c.src[r2.ID], w1.ID) {
return []int{w2, r.ID, r2.ID}, true
}
}
}
return nil, false
}
Causal consistency forbids one thing: a read that returns a value its own causal past has overwritten.
| event | result | why it stays correct | saved by |
|---|---|---|---|
| A user reads a follower right after a write | The follower may not have it. | The token makes the follower wait, or the read goes to the leader. | Session token |
| The load balancer sends the next read to another follower | That follower can be further behind. | The token carries the newest position seen, so no read goes back. | Session token |
| A partition cuts the leader from the majority | The leader cannot commit. | Writes on the minority side fail. The majority side elects a leader and goes on. | Quorum |
| An old leader wakes after a pause | It may serve reads it no longer owns. | It checks with a majority, or its lease expired before the new leader started. | Lease, quorum read |
| Two regions write the same key at once | A conflict. | Give each key a home region, or merge with a CRDT. Last writer wins drops one write. | Home region |
| A cache sits in front of a strong store | The cache returns old values. | Reads that must be strong skip the cache. Writes invalidate the key. | Bypass |
| A reply is replicated before its post | A reader sees the reply alone. | The reply carries the post's position. A copy shows it only after it has the post. | Dependency token |
| step | add | it handles | move up when you see |
|---|---|---|---|
| 1 | One leader for all reads and writes, with a standby for failover. | Linearizable for free. About 107,000 primary-key reads a second on one Postgres in the lab. | Reads load the leader. |
| 2 | Followers for reads, with session tokens. Strong reads stay on the leader. | Reads grow with each follower. Users see their own writes and never go back. | A failover pause is not acceptable, or writes come from many places. |
| 3 | A leaderless quorum: N copies, R + W > N for strong reads. | No failover step. Each request picks its own R and W. | Users on several continents need local latency. |
| 4 | Several regions: local reads with session or causal guarantees. | Region loss. Only strong operations pay a round trip to a majority: 35 ms or more. | Top of the ladder. Shard so each key has a home region. |
Local latencies measured in the lab for the replication sheet. Regional figures are floors: great-circle distance at about 200 km per ms in fibre, there and back. Real cables run longer.
I start with one leader that serves everything, so every read is linearizable. I add followers with session tokens for reads, and pay cross-region round trips only for operations that need them.
0 of 10 known
A write of x = 1 ends. Later, another client reads x = 0. Is the history linearizable? Sequential?
One user sees a new price. A second user, who loads the page after the first finished, sees the old price. Which model forbids this?
A user saves a change and reloads. The change is missing. Which guarantee broke, and what are two fixes?
A reply appears before the post it answers. Which guarantee broke?
State CAP precisely.
There is no partition. Why does a multi-region store still trade consistency?
Can a store stay causally consistent during a partition?
Why is it useful that linearizability is local?
Are serializable and linearizable the same?
An old leader pauses, a new one is elected, and the old one wakes. How do you keep its reads linearizable?
- fibre
- Light covers about 200,000 km a second in fibre: 200 km per ms. Refractive index about 1.47.
- US coasts
- Virginia to Oregon is 3,500 km: 17.5 ms one way, 35 ms there and back.
- Atlantic
- Virginia to Dublin is 5,500 km: 27 ms one way, 55 ms there and back.
- Asia
- Virginia to Mumbai is 12,900 km: 64 ms one way, 129 ms there and back.
- majority
- A strong write waits for the nearest majority: one round trip to the closest other replica, at least.
- local
- 0.14 ms per commit on one Postgres; 0.36 ms with a synchronous standby on the same machine.
Distances are great-circle; round trips are floors. Local commits measured in the lab on an 8-core laptop.