Design a leaderboard

Why not a relational table with an index on the score?

The top 10 is cheap on an index: read the first ten entries in order. A rank is not. A plain B-tree index knows the order but not how many entries sit above a given one, so the rank of a median player is a count over five million entries, about 500 ms at ten million entries a second. The budget is 100 ms. A sorted set keeps a count on every pointer of its skip list, so a rank is about 23 steps.

Does ten million players fit in memory?

Easily. At about 100 bytes a member, pointers included, a board of ten million is 1 GB. Memory is not what limits this design; operations are. The global board takes 50 thousand operations a second at the peak against about 40 thousand for one node, and that is where the round goes.

How do you break ties by the earlier time?

Pack both into the one number the sorted set orders by: the score times the seconds in a month, plus the seconds left in the month when the score was set. A higher score always wins, and between equal scores the earlier one has more seconds left, so it ranks higher. The number must fit the 53 bits a double holds exactly, which allows scores up to about 3.47 billion.

What do you do about the global board that every game writes?

Take the reads off it first. The top 10 changes rarely, so each api box keeps a copy for a second; ranks are read from two replicas. The primary then takes only the writes, 16.7 thousand a second, 41.7 percent of a node. At a hundred times the traffic that is no longer enough, and the board splits by score range into about 42 partitions.

What happens when the node holding a board dies?

Ranks keep coming from the replicas. New scores wait in the update stream while a replica is promoted, about 3 seconds, and are applied as soon as it takes over, so none are lost; they are only late. Without a replica, the board is rebuilt from the last snapshot plus the stream since it was taken, about 27 seconds.