System Design
X7

Conflicts and CRDTs

Two replicas accept writes while they cannot talk. When they meet again, they must agree on one state without losing a write that mattered.

Not startedSaved in this browser only.
  1. 1Conflicts come from writes on more than one replica: several leaders, no leader, or offline clients.
  2. 2Last write wins converges by dropping writes. Siblings keep them and make the app merge.
  3. 3A CRDT merge is commutative, associative and idempotent, so any order and any duplicate give one state.
  4. 4A merge cannot refuse an update. Rules like "never below zero" still need coordination.
X7
    A

    Two replicas, one cart

    recorded from the lab simulation
    offline: no messagesstart: both replicas show {milk}replica Aphonereplica Blaptopadd eggst=1001add breadt=1002remove milkt=1003sync✕ last write wins, whole cartKeeps A's cart, t=1003: {eggs}. B's bread is lost.{eggs}✓ OR-SetMerges each add and each remove: every intent is kept.{bread, eggs}
    • Both replicas accepted every write. Neither saw the other's writes.
    • Concurrent: neither write happened before the other.
    • The merge decides what the user keeps.

    Last write wins always converges, but it converges by dropping a write. A cart needs a type that merges adds and removes.

    B

    Where conflicts come from

    more than one replica accepts writes
    setupwhy two writes are concurrentconflicts
    Single leaderOne node orders every write for a key.None
    Multi-leader, one per regionEach region accepts writes and replicates them later, asynchronously.Yes
    Leaderless, quorum writesAny replica accepts a write. Two writes reach the replicas in different orders.Yes
    Offline clientsA phone is a replica that is cut off for hours, then syncs.Yes
    Collaborative editingSeveral people type into one document in the same second.Yes
    Home region per keyEach key has one writing region; others forward their writes.On failover

    A failover that moves a key's home can leave two regions with writes for one key.

    With one leader there is nothing to merge. Conflicts appear when several replicas accept writes for the same key: several leaders, no leader, or a client that works offline.

    C

    Ways to resolve a conflict

    from dropping a write to merging by design
    methodkeeps both writesmerge logicwhere you meet itstatus
    Avoid them: one writer per keyno conflict to resolvenoneSingle leaderApproved
    Last write wins, by timestampno: one is droppednoneCassandra, DynamoDBLoss is acceptable
    Version vectors, keep siblingsyes, until readthe app merges siblings on readRiak, DynamoApp merge is correct
    Custom resolver per entityas the code decidesa function per entity typeServiceTested for every pair
    Ask the useryesa conflict screen or a conflicted copyServiceRare conflicts
    Operational transformation (OT)yestransform each edit against concurrent editsCentral serverOne server orders edits
    CRDTyes, by the type's rulebuilt into the data typeLibrary or storeApproved

    A version vector holds, per replica, the number of that replica's writes seen. If neither vector is at or below the other, the writes are concurrent.

    I avoid conflicts with one writer per key where I can. Where I cannot, I use last write wins only for data I can lose, and a CRDT or an app merge for the rest.

    D

    Try it: replay the updates

    recorded from the lab simulation
    scenario
    replicate with
    delivery order

    Both replicas start with {milk}. Offline, A adds eggs and removes milk; B adds bread. The whole cart is one value. The write with the larger timestamp wins. Each message reaches the other replicas in the order it was sent.

    step 6 of 6: A2 → B
    1. start
    replica A
    {eggs}
    written by A at 1003
    replica Bmerged
    {eggs}
    written by A at 1003
    All replicas show {eggs}. B’s bread is lost, because A’s cart has the later timestamp.

    Every state is recorded from the lab simulation. The same scenario and order always give the same run.

    A CRDT gives the same state on every replica for any delivery order, even with duplicates. Last write wins also agrees, but on a state that lost writes.

    E

    State-based and operation-based

    two ways to ship an update
    State-based (and delta-state)replica A{A:2, B:0}replica B{A:1, B:3}merge: max per entry{A:2, B:3} = 5✓ any order, any number of copiesThe message grows with the state; a delta stays small.Operation-basedreplica A4replica B4 + 1 = 5+1+1 arrives twice6 ✕⚠ needs exactly-once, causal deliveryMessages stay small; each must arrive once.A merge must have three lawscommutativea ⊔ b = b ⊔ aMessages can arrive in any order.associative(a ⊔ b) ⊔ c = a ⊔ (b ⊔ c)Replicas can merge in any grouping, by gossip.idempotenta ⊔ a = aA duplicate or a retry changes nothing.
    • A delta is the small piece of state that one update changed. It merges like the full state.
    • The lab tests the three laws on every pair and triple of states from 20 seeded histories.
    • It also delivers 7 updates in all 5,040 orders, once and twice each: one final state.

    A state-based CRDT ships state or a delta and merges with a join, so the network can reorder and repeat messages. An operation-based CRDT ships small operations but needs exactly-once, causal delivery.

    F

    The CRDT types

    what each one keeps and what it is for
    typestatemergeuse for
    G-Counterone count per replicamax per entryviews, likes
    PN-Countertwo G-Countersmax per entry, eachcounts that go down
    LWW-Registervalue, timestamp, replicalarger timestampa profile field, last seen
    MV-Registersiblings with vectorskeep the undominatedvalues the app merges
    OR-Settagged adds, tombstonesunion of bothcarts, tags, members
    Map of CRDTsa CRDT per keymerge per keya JSON document
    Sequence: RGA, Yjs, Automergecharacters with IDsunion, then a fixed ordercollaborative text

    Yjs adapts the YATA algorithm. Early Automerge used RGA, the algorithm this sheet's lab implements.

    I pick the type by the operation. Counts get a counter, membership an OR-Set, one value a register, and text a sequence CRDT.

    G

    Counters

    pseudo code
    G-Counter and PN-Counterpseudo code
    G-Counter: one entry per replica
      inc(r, n):     state[r] += n              // only r writes entry r1
      value():       sum of all entries
      merge(a, b):   FOR EACH replica r: max(a[r], b[r])2
    
    PN-Counter: two G-Counters3
      add(r, n):     IF n >= 0: P.inc(r, n)  ELSE: N.inc(r, -n)
      value():       P.value() - N.value()
      merge(a, b):   (merge(a.P, b.P), merge(a.N, b.N))
    1. 1Each replica owns its entry, so two replicas never write the same entry.
    2. 2An entry only grows, so the larger one has seen more increments. Max is commutative, associative and idempotent.
    3. 3A decrement would shrink an entry, and max would undo it. Decrements go into a second counter instead.
    Tested source Go: counters
    Go: countersgo
    
    // Inc adds n (n >= 0) to replica r's own entry. It returns the new state and the delta to send.
    func (g GCounter) Inc(r string, n int64) (GCounter, GCounter) {
      if n < 0 {
        panic(fmt.Sprintf("GCounter.Inc: negative amount %d", n))
      }
      out := maps.Clone(g)
      if out == nil {
        out = GCounter{}
      }
      out[r] += n
      return out, GCounter{r: out[r]}
    }
    
    // Value is the sum of every replica's entry.
    func (g GCounter) Value() int64 {
      var v int64
      for _, n := range g {
        v += n
      }
      return v
    }
    
    // Merge takes the larger entry for each replica. An entry only grows, so the larger one has
    // seen more of that replica's increments.
    func (g GCounter) Merge(o GCounter) GCounter {
      out := maps.Clone(g)
      if out == nil {
        out = GCounter{}
      }
      for r, n := range o {
        out[r] = max(out[r], n)
      }
      return out
    }
    
    // PNCounter counts up and down with two grow-only counters: increments and decrements.
    type PNCounter struct {
      P GCounter `json:"p"`
      N GCounter `json:"n"`
    }
    
    // Add adds n, which may be negative, at replica r.
    func (c PNCounter) Add(r string, n int64) (PNCounter, PNCounter) {
      if n >= 0 {
        p, d := c.P.Inc(r, n)
        return PNCounter{P: p, N: c.N}, PNCounter{P: d}
      }
      m, d := c.N.Inc(r, -n)
      return PNCounter{P: c.P, N: m}, PNCounter{N: d}
    }
    
    // Value is increments minus decrements.
    func (c PNCounter) Value() int64 { return c.P.Value() - c.N.Value() }
    
    // Merge merges both halves.
    func (c PNCounter) Merge(o PNCounter) PNCounter {
      return PNCounter{P: c.P.Merge(o.P), N: c.N.Merge(o.N)}
    }
    

    Cost: one entry per replica that ever wrote. 1,000 writers keep 1,000 entries in every copy.

    Each replica owns one entry in the counter, and merge takes the larger entry. To count down, I keep a second counter for decrements.

    H

    Registers

    pseudo code
    LWW and multi-value registerspseudo code
    LWW-Register: (value, ts, replica)
      set(r, v):     (v, clock of r1, r)
      merge(a, b):   keep the larger (ts, replica)   // drop the other2
    
    MV-Register: siblings, each (value, version vector)
      set(r, v):     vv = join of all sibling vectors3; vv[r] += 1
                     siblings = [(v, vv)]             // replaces what r has seen
      merge(a, b):   keep each sibling that no other vector dominates4
      read():        every sibling; the app merges them5
    1. 1A wall clock. Under skew, the later write in real time can carry the smaller stamp and lose.
    2. 2Every replica drops the same write, so they converge. The dropped write is gone.
    3. 3The new write has seen every sibling here, so it replaces them all.
    4. 4Concurrent writes survive side by side. Dynamo calls them siblings.
    5. 5For a cart, Dynamo took the union. A removed item can then come back.
    Tested source Go: registers
    Go: registersgo
    
    // Set writes v at replica r with timestamp ts from r's clock.
    func (l LWW[T]) Set(r string, v T, ts int64) (LWW[T], LWW[T]) {
      w := LWW[T]{Value: v, TS: ts, Replica: r}
      return l.Merge(w), w
    }
    
    // Merge keeps the later write. Equal timestamps fall back to the replica name, so every
    // replica picks the same winner. The loser is gone, even if it happened later in real time.
    func (l LWW[T]) Merge(o LWW[T]) LWW[T] {
      if o.TS > l.TS || (o.TS == l.TS && o.Replica > l.Replica) {
        return o
      }
      return l
    }
    
    // MV is a multi-value register. Each write carries a version vector. A write that has seen
    // another write replaces it; two concurrent writes are both kept, as siblings, until a later
    // write that has seen both replaces them.
    type MV[T comparable] struct {
      Entries []MVEntry[T] `json:"entries"`
    }
    
    // MVEntry is one sibling: a value and the version vector of its write.
    type MVEntry[T comparable] struct {
      Value T  `json:"value"`
      VV    VV `json:"vv"`
    }
    
    // Set writes v at replica r. Its vector is the join of every sibling's vector plus one for r,
    // so it supersedes every sibling this replica has seen.
    func (m MV[T]) Set(r string, v T) (MV[T], MV[T]) {
      vv := VV{}
      for _, e := range m.Entries {
        vv = vv.Join(e.VV)
      }
      vv[r]++
      w := MV[T]{Entries: []MVEntry[T]{{Value: v, VV: vv}}}
      return m.Merge(w), w
    }
    
    // Merge keeps every entry that no other entry's vector dominates.
    func (m MV[T]) Merge(o MV[T]) MV[T] {
      all := append(slices.Clone(m.Entries), o.Entries...)
      var out []MVEntry[T]
      for i, e := range all {
        keep := true
        for j, f := range all {
          if i != j && (e.VV.Before(f.VV) || (e.VV.Equal(f.VV) && j < i)) {
            keep = false // dominated, or a duplicate of an earlier entry
            break
          }
        }
        if keep {
          out = append(out, e)
        }
      }
      sortEntries(out)
      return MV[T]{Entries: out}
    }
    

    A last-write-wins register keeps one write and drops the rest. A multi-value register keeps concurrent writes as siblings and lets the app merge them.

    I

    Observed-remove set

    pseudo code
    OR-Setpseudo code
    add(r, e):      tag = (r, next counter of r1)
                    adds[e] += tag
    remove(e):      tombstones += adds[e]      // only the tags seen here2
    has(e):         some tag of e is not a tombstone
    merge(a, b):    (adds of a ∪ adds of b, tombstones of a ∪ tombstones of b3)
    1. 1Replica and counter make the tag unique. Adding milk again gives a new tag.
    2. 2A concurrent add elsewhere has a tag this replica has not seen, so it survives.
    3. 3Unions only grow, so the merge is a join. A remove that arrives before its add still applies.
    Tested source Go: OR-Set
    Go: OR-Setgo
    
    // Add adds e at replica r with a new tag.
    func (s ORSet) Add(r, e string) (ORSet, ORSet) {
      t := Tag{R: r, N: s.lastTag(r) + 1}
      d := ORSet{Adds: map[string]map[Tag]bool{e: {t: true}}}
      return s.Merge(d), d
    }
    
    // Remove removes e: it tombstones every tag of e that this replica has observed.
    func (s ORSet) Remove(e string) (ORSet, ORSet) {
      d := ORSet{Removed: map[Tag]bool{}}
      for t := range s.Adds[e] {
        if !s.Removed[t] {
          d.Removed[t] = true
        }
      }
      return s.Merge(d), d
    }
    
    // Has reports whether e has a tag that no remove has observed.
    func (s ORSet) Has(e string) bool {
      for t := range s.Adds[e] {
        if !s.Removed[t] {
          return true
        }
      }
      return false
    }
    
    // Merge is the union of the adds and the union of the tombstones.
    func (s ORSet) Merge(o ORSet) ORSet {
      out := ORSet{Adds: map[string]map[Tag]bool{}, Removed: map[Tag]bool{}}
      for _, x := range []ORSet{s, o} {
        for e, tags := range x.Adds {
          if out.Adds[e] == nil {
            out.Adds[e] = map[Tag]bool{}
          }
          maps.Copy(out.Adds[e], tags)
        }
        maps.Copy(out.Removed, x.Removed)
      }
      return out
    }
    

    After 2,000 random adds and removes on 100 items, the set keeps 967 tags for 52 items.

    Each add in an OR-Set gets a unique tag, and a remove deletes only the tags it has seen. So an add that the remover never saw survives: add wins.

    J

    Collaborative text: RGA

    pseudo code, then the tree
    RGA textpseudo code
    insert(r, pos, s):
      after = ID of the visible character at pos - 1, or root
      FOR EACH character ch IN s:
        id = (largest counter seen + 11, r)
        add node (id, after, ch); after = id
    delete(pos, n):  mark n visible nodes as tombstones2
    merge(a, b):     union of nodes, union of tombstones
    read():          depth first from root, newest child first3;
                     skip tombstones; a node waits for its anchor4
    1. 1The new ID is above every ID this replica knows, so it is the newest child of its anchor.
    2. 2The node stays as an anchor for concurrent inserts.
    3. 3Two inserts at one place come out in the same order everywhere. Each word stays whole.
    4. 4Out-of-order delivery is safe: the node appears once its anchor arrives.
    Tested source Go: RGA text
    Go: RGA textgo
    
    // Insert inserts s at visible position pos, at replica r. Each character gets an ID above every
    // ID this replica has seen, and hangs under the character before it.
    func (t Text) Insert(r string, pos int, s string) (Text, Text) {
      after := root
      if pos > 0 {
        after = t.visible()[pos-1]
      }
      c := t.maxCounter()
      d := Text{Chars: map[ID]Char{}}
      for _, l := range s {
        c++
        id := ID{C: c, R: r}
        d.Chars[id] = Char{ID: id, After: after, Letter: string(l)}
        after = id
      }
      return t.Merge(d), d
    }
    
    // Delete marks n visible characters from pos as tombstones.
    func (t Text) Delete(pos, n int) (Text, Text) {
      d := Text{Deleted: map[ID]bool{}}
      for _, id := range t.visible()[pos : pos+n] {
        d.Deleted[id] = true
      }
      return t.Merge(d), d
    }
    
    // Merge is the union of the characters and the union of the tombstones.
    func (t Text) Merge(o Text) Text {
      out := Text{Chars: map[ID]Char{}, Deleted: map[ID]bool{}}
      for _, x := range []Text{t, o} {
        maps.Copy(out.Chars, x.Chars)
        maps.Copy(out.Deleted, x.Deleted)
      }
      return out
    }
    
    // walk visits the tree depth first, children newest first. A character whose anchor has not
    // arrived yet is not reachable, so it stays hidden until the anchor arrives.
    func (t Text) walk(visit func(Char)) {
      kids := map[ID][]Char{}
      for _, c := range t.Chars {
        kids[c.After] = append(kids[c.After], c)
      }
      var down func(ID)
      down = func(at ID) {
        ks := kids[at]
        slices.SortFunc(ks, func(a, b Char) int { return b.ID.compare(a.ID) })
        for _, k := range ks {
          visit(k)
          down(k.ID)
        }
      }
      down(root)
    }
    
    rootb1Au2Ay3A␣4Ao9Aa10At11A␣12Am5Ai6Al7Ak8A!9Bnewer childread firstRead depth first, newest child first; skip tombstones.every replica reads"oat milk!"

    A sequence CRDT gives every character an ID and an anchor, so an insert never depends on a position. Deleted characters stay as tombstones, because other inserts may still refer to them.

    K

    OT or a CRDT

    for collaborative editing
    questionOTsequence CRDT
    An edit refers toa positiona character's ID
    Concurrent editstransform positionsmerge sets of nodes
    Needs a central serverIn practiceNo
    Works offline, peer to peerHardYes
    Metadata per characterNoneID and tombstones
    • Replayed without a transform, the positions in the lab diverge in every delivery order.
    • After 2,000 random edits, the RGA keeps 1,392 characters to show 784.

    Operational transformation rewrites each edit's position against concurrent edits, which needs one server to order edits. A CRDT gives each character an identity, so any replica can merge without a server.

    L

    What a CRDT cannot do

    recorded from the lab simulation
    rule: balance never below 0 · start 100 on both replicasreplica A: withdraw 40check 100 ≥ 40✓ passeslocal balance 60replica B: withdraw 80check 100 ≥ 80✓ passeslocal balance 20merge both updates✕ PN-Counter100 − 40 − 80. Both replicas converge, below zero.-20⚠ escrow: each replica owns half the rightsB may spend only 50, so it refuses 80. To spend more,B must ask A for rights: that is coordination.60
    • Uniqueness, such as one owner per username, also needs one decision point.
    • Escrow keeps the rule without a round trip, until a replica runs out of rights.

    A CRDT merge cannot refuse an update, so it cannot keep a rule like 'balance never below zero'. For that I route the write through one leader or give each replica an escrow share.

    M

    Capabilities used

    what each tool gives you
    toolcapabilitywhat it gives this designalso used for
    PostgresOne primary orders every writeNo conflicts: a transaction sees the latest committed row.Anything with an invariant
    PostgresLogical replication in both directionsNo merge An incoming change overwrites the row. A constraint violation stops replication until someone fixes it.Region migrations
    Redis Active-ActiveCRDT-based replication between clustersCounters add up across regions; sets keep a concurrent add; strings take the last write.Sessions, leaderboards across regions
    ServiceCRDT types in a library, state in a document columnMerge carts and counters in the service, with any store behind it.Offline sync for mobile apps
    ServiceEscrow rights per replicaKeeps a bound such as stock or balance without a round trip per write.Ticket and stock quotas
    CassandraLast write wins per cell, by write timestamp; counter columnsFast multi-region writes for data that can lose a concurrent write.Time series, user activity
    DynamoDBGlobal tables: last writer wins between RegionsLoss accepted Concurrent writes to one item keep one.Multi-region tables
    Riak KVSiblings with version vectors; counters, sets, mapsKeeps concurrent writes, or merges them with a built-in type.Carts, profiles
    Yjs, AutomergeSequence and map CRDTs with a sync protocolCollaborative text and JSON that merge offline or peer to peer.Local-first apps

    One Postgres primary has no write conflicts at all. Across regions, I choose per data type. Disposable fields get last write wins; counters and sets get CRDT types; shared text gets a sequence CRDT.

    N

    Failure cases

    what breaks, and what saves you
    eventresultwhy it stays correctsaved by
    A phone edits a cart offline for two daysIts adds and removes meet the laptop's.The OR-Set merges each add and remove; a concurrent add wins over a remove.OR-Set
    A sync message is sent again after a timeoutThe replica receives it twice.Merge is idempotent: the second copy changes nothing.State merge
    Updates arrive out of orderReplicas pass through different states.Merge is commutative; an RGA node waits until its anchor arrives.RGA
    Two regions write one key; clocks are skewedLast write wins drops the newer write.Use last write wins only where a lost write is acceptable.Type per field
    Clients write without the vector they readSiblings pile up on every write.The client sends the context it read, so a write replaces what it saw.Riak contexts
    Tombstones grow without limitState and sync size grow.Drop a tombstone once every replica has seen it, or compact to a snapshot.Garbage collection
    Two replicas each approve a withdrawalThe merged balance is below zero.Withdrawals go through one leader, or spend only local escrow rights.Single leader
    O

    Scale ladder

    start simple; climb only on a signal
    Each step adds one component1One primary2+ home region3+ LWW fields4+ CRDT types5+ text CRDTmore load →
    Metadata kept after 2,000 updates1101001k10kLWW register: 1 entries keptLWW register1G-Counter, 3 writers: 3 entries keptG-Counter, 3 writers3G-Counter, 1,000 writers: 1,000 entries keptG-Counter, 1,000 writers1,000OR-Set, 52 items: 967 entries keptOR-Set, 52 items967RGA, 784 chars: 1,392 entries keptRGA, 784 chars1,392entries kept, log scale
    stepaddit handlesmove up when you see
    1One primary for all writes, replicas for reads.No conflicts. Every invariant holds in one transaction.Users far from the primary wait one long round trip per write.
    2A home region per user, which takes all of that user's writes.Local writes for most users, still with one writer per key.One record gets writes from two regions, or a failover moves the home.
    3Last write wins for fields where losing a write is acceptable.Profile fields, "last seen", settings. Simple and small.Users notice lost adds, likes or edits.
    4CRDT types for counters, sets and carts, plus escrow for bounds.Concurrent writes in every region and on offline clients, with every intent kept.People edit the same text together, or the app must work offline first.
    5A sequence CRDT for text and lists, with a sync server.Real-time and offline co-editing, peer to peer if needed.Top of the ladder. Watch tombstone growth.

    Metadata counted in the lab: tags for the OR-Set, characters kept for the RGA, entries for the counter. Do not start at step 4. A CRDT never removes the need for one decision point for invariants.

    I keep one writer per key for as long as I can. I add CRDT types only for the data that must accept writes in several places, and a sequence CRDT only for shared editing.

    P

    Drill

    predict, then reveal

    0 of 9 known

    1. A phone and a laptop each add an item to one cart while offline. The cart is one last-write-wins value. What survives?

    2. Dynamo merges cart siblings by union. Why can a deleted item come back?

    3. Which three laws must a merge obey, and what does each one buy you?

    4. An operation-based counter sends "add 1". The network delivers one message twice. What happens?

    5. Why can a G-Counter not count down, and how does a PN-Counter fix it?

    6. Replica A removes milk while replica B adds milk at the same time. What does an OR-Set show?

    7. Why does RGA keep deleted characters?

    8. A balance must never go below zero. Can a CRDT keep that rule?

    9. Why do most systems with operational transformation use a central server?

    Q

    Numbers to say

    measured, derived or cited
    orders
    7 updates have 7! = 5,040 delivery orders. Every type ends in one state in all of them.
    cart
    Last write wins lost 1 of 3 offline updates; the OR-Set lost none.
    counter
    One entry per writer: 1,000 writers keep 1,000 entries in every copy.
    OR-Set
    967 tags kept for 52 items after 2,000 updates.
    RGA
    1,392 characters kept to show 784 after 2,000 edits.
    vector
    One counter per replica in every version vector.

    Lab: seeded simulation, so the counts are exact. Dynamo's cart behaviour: the Dynamo paper, SOSP 2007.