Conflicts and CRDTs
Two replicas accept writes while they cannot talk. When they meet again, they must agree on one state without losing a write that mattered.
- 1Conflicts come from writes on more than one replica: several leaders, no leader, or offline clients.
- 2Last write wins converges by dropping writes. Siblings keep them and make the app merge.
- 3A CRDT merge is commutative, associative and idempotent, so any order and any duplicate give one state.
- 4A merge cannot refuse an update. Rules like "never below zero" still need coordination.
- Both replicas accepted every write. Neither saw the other's writes.
- Concurrent: neither write happened before the other.
- The merge decides what the user keeps.
Last write wins always converges, but it converges by dropping a write. A cart needs a type that merges adds and removes.
| setup | why two writes are concurrent | conflicts |
|---|---|---|
| Single leader | One node orders every write for a key. | None |
| Multi-leader, one per region | Each region accepts writes and replicates them later, asynchronously. | Yes |
| Leaderless, quorum writes | Any replica accepts a write. Two writes reach the replicas in different orders. | Yes |
| Offline clients | A phone is a replica that is cut off for hours, then syncs. | Yes |
| Collaborative editing | Several people type into one document in the same second. | Yes |
| Home region per key | Each key has one writing region; others forward their writes. | On failover |
A failover that moves a key's home can leave two regions with writes for one key.
With one leader there is nothing to merge. Conflicts appear when several replicas accept writes for the same key: several leaders, no leader, or a client that works offline.
| method | keeps both writes | merge logic | where you meet it | status |
|---|---|---|---|---|
| Avoid them: one writer per key | no conflict to resolve | none | Single leader | Approved |
| Last write wins, by timestamp | no: one is dropped | none | Cassandra, DynamoDB | Loss is acceptable |
| Version vectors, keep siblings | yes, until read | the app merges siblings on read | Riak, Dynamo | App merge is correct |
| Custom resolver per entity | as the code decides | a function per entity type | Service | Tested for every pair |
| Ask the user | yes | a conflict screen or a conflicted copy | Service | Rare conflicts |
| Operational transformation (OT) | yes | transform each edit against concurrent edits | Central server | One server orders edits |
| CRDT | yes, by the type's rule | built into the data type | Library or store | Approved |
A version vector holds, per replica, the number of that replica's writes seen. If neither vector is at or below the other, the writes are concurrent.
I avoid conflicts with one writer per key where I can. Where I cannot, I use last write wins only for data I can lose, and a CRDT or an app merge for the rest.
Both replicas start with {milk}. Offline, A adds eggs and removes milk; B adds bread. The whole cart is one value. The write with the larger timestamp wins. Each message reaches the other replicas in the order it was sent.
- start
Every state is recorded from the lab simulation. The same scenario and order always give the same run.
A CRDT gives the same state on every replica for any delivery order, even with duplicates. Last write wins also agrees, but on a state that lost writes.
- A delta is the small piece of state that one update changed. It merges like the full state.
- The lab tests the three laws on every pair and triple of states from 20 seeded histories.
- It also delivers 7 updates in all 5,040 orders, once and twice each: one final state.
A state-based CRDT ships state or a delta and merges with a join, so the network can reorder and repeat messages. An operation-based CRDT ships small operations but needs exactly-once, causal delivery.
| type | state | merge | use for |
|---|---|---|---|
| G-Counter | one count per replica | max per entry | views, likes |
| PN-Counter | two G-Counters | max per entry, each | counts that go down |
| LWW-Register | value, timestamp, replica | larger timestamp | a profile field, last seen |
| MV-Register | siblings with vectors | keep the undominated | values the app merges |
| OR-Set | tagged adds, tombstones | union of both | carts, tags, members |
| Map of CRDTs | a CRDT per key | merge per key | a JSON document |
| Sequence: RGA, Yjs, Automerge | characters with IDs | union, then a fixed order | collaborative text |
Yjs adapts the YATA algorithm. Early Automerge used RGA, the algorithm this sheet's lab implements.
I pick the type by the operation. Counts get a counter, membership an OR-Set, one value a register, and text a sequence CRDT.
G-Counter: one entry per replica
inc(r, n): state[r] += n // only r writes entry r1
value(): sum of all entries
merge(a, b): FOR EACH replica r: max(a[r], b[r])2
PN-Counter: two G-Counters3
add(r, n): IF n >= 0: P.inc(r, n) ELSE: N.inc(r, -n)
value(): P.value() - N.value()
merge(a, b): (merge(a.P, b.P), merge(a.N, b.N))- 1Each replica owns its entry, so two replicas never write the same entry.
- 2An entry only grows, so the larger one has seen more increments. Max is commutative, associative and idempotent.
- 3A decrement would shrink an entry, and max would undo it. Decrements go into a second counter instead.
Tested source Go: counters
// Inc adds n (n >= 0) to replica r's own entry. It returns the new state and the delta to send.
func (g GCounter) Inc(r string, n int64) (GCounter, GCounter) {
if n < 0 {
panic(fmt.Sprintf("GCounter.Inc: negative amount %d", n))
}
out := maps.Clone(g)
if out == nil {
out = GCounter{}
}
out[r] += n
return out, GCounter{r: out[r]}
}
// Value is the sum of every replica's entry.
func (g GCounter) Value() int64 {
var v int64
for _, n := range g {
v += n
}
return v
}
// Merge takes the larger entry for each replica. An entry only grows, so the larger one has
// seen more of that replica's increments.
func (g GCounter) Merge(o GCounter) GCounter {
out := maps.Clone(g)
if out == nil {
out = GCounter{}
}
for r, n := range o {
out[r] = max(out[r], n)
}
return out
}
// PNCounter counts up and down with two grow-only counters: increments and decrements.
type PNCounter struct {
P GCounter `json:"p"`
N GCounter `json:"n"`
}
// Add adds n, which may be negative, at replica r.
func (c PNCounter) Add(r string, n int64) (PNCounter, PNCounter) {
if n >= 0 {
p, d := c.P.Inc(r, n)
return PNCounter{P: p, N: c.N}, PNCounter{P: d}
}
m, d := c.N.Inc(r, -n)
return PNCounter{P: c.P, N: m}, PNCounter{N: d}
}
// Value is increments minus decrements.
func (c PNCounter) Value() int64 { return c.P.Value() - c.N.Value() }
// Merge merges both halves.
func (c PNCounter) Merge(o PNCounter) PNCounter {
return PNCounter{P: c.P.Merge(o.P), N: c.N.Merge(o.N)}
}
Cost: one entry per replica that ever wrote. 1,000 writers keep 1,000 entries in every copy.
Each replica owns one entry in the counter, and merge takes the larger entry. To count down, I keep a second counter for decrements.
LWW-Register: (value, ts, replica)
set(r, v): (v, clock of r1, r)
merge(a, b): keep the larger (ts, replica) // drop the other2
MV-Register: siblings, each (value, version vector)
set(r, v): vv = join of all sibling vectors3; vv[r] += 1
siblings = [(v, vv)] // replaces what r has seen
merge(a, b): keep each sibling that no other vector dominates4
read(): every sibling; the app merges them5- 1A wall clock. Under skew, the later write in real time can carry the smaller stamp and lose.
- 2Every replica drops the same write, so they converge. The dropped write is gone.
- 3The new write has seen every sibling here, so it replaces them all.
- 4Concurrent writes survive side by side. Dynamo calls them siblings.
- 5For a cart, Dynamo took the union. A removed item can then come back.
Tested source Go: registers
// Set writes v at replica r with timestamp ts from r's clock.
func (l LWW[T]) Set(r string, v T, ts int64) (LWW[T], LWW[T]) {
w := LWW[T]{Value: v, TS: ts, Replica: r}
return l.Merge(w), w
}
// Merge keeps the later write. Equal timestamps fall back to the replica name, so every
// replica picks the same winner. The loser is gone, even if it happened later in real time.
func (l LWW[T]) Merge(o LWW[T]) LWW[T] {
if o.TS > l.TS || (o.TS == l.TS && o.Replica > l.Replica) {
return o
}
return l
}
// MV is a multi-value register. Each write carries a version vector. A write that has seen
// another write replaces it; two concurrent writes are both kept, as siblings, until a later
// write that has seen both replaces them.
type MV[T comparable] struct {
Entries []MVEntry[T] `json:"entries"`
}
// MVEntry is one sibling: a value and the version vector of its write.
type MVEntry[T comparable] struct {
Value T `json:"value"`
VV VV `json:"vv"`
}
// Set writes v at replica r. Its vector is the join of every sibling's vector plus one for r,
// so it supersedes every sibling this replica has seen.
func (m MV[T]) Set(r string, v T) (MV[T], MV[T]) {
vv := VV{}
for _, e := range m.Entries {
vv = vv.Join(e.VV)
}
vv[r]++
w := MV[T]{Entries: []MVEntry[T]{{Value: v, VV: vv}}}
return m.Merge(w), w
}
// Merge keeps every entry that no other entry's vector dominates.
func (m MV[T]) Merge(o MV[T]) MV[T] {
all := append(slices.Clone(m.Entries), o.Entries...)
var out []MVEntry[T]
for i, e := range all {
keep := true
for j, f := range all {
if i != j && (e.VV.Before(f.VV) || (e.VV.Equal(f.VV) && j < i)) {
keep = false // dominated, or a duplicate of an earlier entry
break
}
}
if keep {
out = append(out, e)
}
}
sortEntries(out)
return MV[T]{Entries: out}
}
A last-write-wins register keeps one write and drops the rest. A multi-value register keeps concurrent writes as siblings and lets the app merge them.
add(r, e): tag = (r, next counter of r1)
adds[e] += tag
remove(e): tombstones += adds[e] // only the tags seen here2
has(e): some tag of e is not a tombstone
merge(a, b): (adds of a ∪ adds of b, tombstones of a ∪ tombstones of b3)- 1Replica and counter make the tag unique. Adding milk again gives a new tag.
- 2A concurrent add elsewhere has a tag this replica has not seen, so it survives.
- 3Unions only grow, so the merge is a join. A remove that arrives before its add still applies.
Tested source Go: OR-Set
// Add adds e at replica r with a new tag.
func (s ORSet) Add(r, e string) (ORSet, ORSet) {
t := Tag{R: r, N: s.lastTag(r) + 1}
d := ORSet{Adds: map[string]map[Tag]bool{e: {t: true}}}
return s.Merge(d), d
}
// Remove removes e: it tombstones every tag of e that this replica has observed.
func (s ORSet) Remove(e string) (ORSet, ORSet) {
d := ORSet{Removed: map[Tag]bool{}}
for t := range s.Adds[e] {
if !s.Removed[t] {
d.Removed[t] = true
}
}
return s.Merge(d), d
}
// Has reports whether e has a tag that no remove has observed.
func (s ORSet) Has(e string) bool {
for t := range s.Adds[e] {
if !s.Removed[t] {
return true
}
}
return false
}
// Merge is the union of the adds and the union of the tombstones.
func (s ORSet) Merge(o ORSet) ORSet {
out := ORSet{Adds: map[string]map[Tag]bool{}, Removed: map[Tag]bool{}}
for _, x := range []ORSet{s, o} {
for e, tags := range x.Adds {
if out.Adds[e] == nil {
out.Adds[e] = map[Tag]bool{}
}
maps.Copy(out.Adds[e], tags)
}
maps.Copy(out.Removed, x.Removed)
}
return out
}
After 2,000 random adds and removes on 100 items, the set keeps 967 tags for 52 items.
Each add in an OR-Set gets a unique tag, and a remove deletes only the tags it has seen. So an add that the remover never saw survives: add wins.
insert(r, pos, s):
after = ID of the visible character at pos - 1, or root
FOR EACH character ch IN s:
id = (largest counter seen + 11, r)
add node (id, after, ch); after = id
delete(pos, n): mark n visible nodes as tombstones2
merge(a, b): union of nodes, union of tombstones
read(): depth first from root, newest child first3;
skip tombstones; a node waits for its anchor4- 1The new ID is above every ID this replica knows, so it is the newest child of its anchor.
- 2The node stays as an anchor for concurrent inserts.
- 3Two inserts at one place come out in the same order everywhere. Each word stays whole.
- 4Out-of-order delivery is safe: the node appears once its anchor arrives.
Tested source Go: RGA text
// Insert inserts s at visible position pos, at replica r. Each character gets an ID above every
// ID this replica has seen, and hangs under the character before it.
func (t Text) Insert(r string, pos int, s string) (Text, Text) {
after := root
if pos > 0 {
after = t.visible()[pos-1]
}
c := t.maxCounter()
d := Text{Chars: map[ID]Char{}}
for _, l := range s {
c++
id := ID{C: c, R: r}
d.Chars[id] = Char{ID: id, After: after, Letter: string(l)}
after = id
}
return t.Merge(d), d
}
// Delete marks n visible characters from pos as tombstones.
func (t Text) Delete(pos, n int) (Text, Text) {
d := Text{Deleted: map[ID]bool{}}
for _, id := range t.visible()[pos : pos+n] {
d.Deleted[id] = true
}
return t.Merge(d), d
}
// Merge is the union of the characters and the union of the tombstones.
func (t Text) Merge(o Text) Text {
out := Text{Chars: map[ID]Char{}, Deleted: map[ID]bool{}}
for _, x := range []Text{t, o} {
maps.Copy(out.Chars, x.Chars)
maps.Copy(out.Deleted, x.Deleted)
}
return out
}
// walk visits the tree depth first, children newest first. A character whose anchor has not
// arrived yet is not reachable, so it stays hidden until the anchor arrives.
func (t Text) walk(visit func(Char)) {
kids := map[ID][]Char{}
for _, c := range t.Chars {
kids[c.After] = append(kids[c.After], c)
}
var down func(ID)
down = func(at ID) {
ks := kids[at]
slices.SortFunc(ks, func(a, b Char) int { return b.ID.compare(a.ID) })
for _, k := range ks {
visit(k)
down(k.ID)
}
}
down(root)
}
A sequence CRDT gives every character an ID and an anchor, so an insert never depends on a position. Deleted characters stay as tombstones, because other inserts may still refer to them.
| question | OT | sequence CRDT |
|---|---|---|
| An edit refers to | a position | a character's ID |
| Concurrent edits | transform positions | merge sets of nodes |
| Needs a central server | In practice | No |
| Works offline, peer to peer | Hard | Yes |
| Metadata per character | None | ID and tombstones |
- Replayed without a transform, the positions in the lab diverge in every delivery order.
- After 2,000 random edits, the RGA keeps 1,392 characters to show 784.
Operational transformation rewrites each edit's position against concurrent edits, which needs one server to order edits. A CRDT gives each character an identity, so any replica can merge without a server.
- Uniqueness, such as one owner per username, also needs one decision point.
- Escrow keeps the rule without a round trip, until a replica runs out of rights.
A CRDT merge cannot refuse an update, so it cannot keep a rule like 'balance never below zero'. For that I route the write through one leader or give each replica an escrow share.
| tool | capability | what it gives this design | also used for |
|---|---|---|---|
| Postgres | One primary orders every write | No conflicts: a transaction sees the latest committed row. | Anything with an invariant |
| Postgres | Logical replication in both directions | No merge An incoming change overwrites the row. A constraint violation stops replication until someone fixes it. | Region migrations |
| Redis Active-Active | CRDT-based replication between clusters | Counters add up across regions; sets keep a concurrent add; strings take the last write. | Sessions, leaderboards across regions |
| Service | CRDT types in a library, state in a document column | Merge carts and counters in the service, with any store behind it. | Offline sync for mobile apps |
| Service | Escrow rights per replica | Keeps a bound such as stock or balance without a round trip per write. | Ticket and stock quotas |
| Cassandra | Last write wins per cell, by write timestamp; counter columns | Fast multi-region writes for data that can lose a concurrent write. | Time series, user activity |
| DynamoDB | Global tables: last writer wins between Regions | Loss accepted Concurrent writes to one item keep one. | Multi-region tables |
| Riak KV | Siblings with version vectors; counters, sets, maps | Keeps concurrent writes, or merges them with a built-in type. | Carts, profiles |
| Yjs, Automerge | Sequence and map CRDTs with a sync protocol | Collaborative text and JSON that merge offline or peer to peer. | Local-first apps |
One Postgres primary has no write conflicts at all. Across regions, I choose per data type. Disposable fields get last write wins; counters and sets get CRDT types; shared text gets a sequence CRDT.
| event | result | why it stays correct | saved by |
|---|---|---|---|
| A phone edits a cart offline for two days | Its adds and removes meet the laptop's. | The OR-Set merges each add and remove; a concurrent add wins over a remove. | OR-Set |
| A sync message is sent again after a timeout | The replica receives it twice. | Merge is idempotent: the second copy changes nothing. | State merge |
| Updates arrive out of order | Replicas pass through different states. | Merge is commutative; an RGA node waits until its anchor arrives. | RGA |
| Two regions write one key; clocks are skewed | Last write wins drops the newer write. | Use last write wins only where a lost write is acceptable. | Type per field |
| Clients write without the vector they read | Siblings pile up on every write. | The client sends the context it read, so a write replaces what it saw. | Riak contexts |
| Tombstones grow without limit | State and sync size grow. | Drop a tombstone once every replica has seen it, or compact to a snapshot. | Garbage collection |
| Two replicas each approve a withdrawal | The merged balance is below zero. | Withdrawals go through one leader, or spend only local escrow rights. | Single leader |
| step | add | it handles | move up when you see |
|---|---|---|---|
| 1 | One primary for all writes, replicas for reads. | No conflicts. Every invariant holds in one transaction. | Users far from the primary wait one long round trip per write. |
| 2 | A home region per user, which takes all of that user's writes. | Local writes for most users, still with one writer per key. | One record gets writes from two regions, or a failover moves the home. |
| 3 | Last write wins for fields where losing a write is acceptable. | Profile fields, "last seen", settings. Simple and small. | Users notice lost adds, likes or edits. |
| 4 | CRDT types for counters, sets and carts, plus escrow for bounds. | Concurrent writes in every region and on offline clients, with every intent kept. | People edit the same text together, or the app must work offline first. |
| 5 | A sequence CRDT for text and lists, with a sync server. | Real-time and offline co-editing, peer to peer if needed. | Top of the ladder. Watch tombstone growth. |
Metadata counted in the lab: tags for the OR-Set, characters kept for the RGA, entries for the counter. Do not start at step 4. A CRDT never removes the need for one decision point for invariants.
I keep one writer per key for as long as I can. I add CRDT types only for the data that must accept writes in several places, and a sequence CRDT only for shared editing.
0 of 9 known
A phone and a laptop each add an item to one cart while offline. The cart is one last-write-wins value. What survives?
Dynamo merges cart siblings by union. Why can a deleted item come back?
Which three laws must a merge obey, and what does each one buy you?
An operation-based counter sends "add 1". The network delivers one message twice. What happens?
Why can a G-Counter not count down, and how does a PN-Counter fix it?
Replica A removes milk while replica B adds milk at the same time. What does an OR-Set show?
Why does RGA keep deleted characters?
A balance must never go below zero. Can a CRDT keep that rule?
Why do most systems with operational transformation use a central server?
- orders
- 7 updates have 7! = 5,040 delivery orders. Every type ends in one state in all of them.
- cart
- Last write wins lost 1 of 3 offline updates; the OR-Set lost none.
- counter
- One entry per writer: 1,000 writers keep 1,000 entries in every copy.
- OR-Set
- 967 tags kept for 52 items after 2,000 updates.
- RGA
- 1,392 characters kept to show 784 after 2,000 edits.
- vector
- One counter per replica in every version vector.
Lab: seeded simulation, so the counts are exact. Dynamo's cart behaviour: the Dynamo paper, SOSP 2007.