Design an S3-like object store
A hundred petabytes in ten billion objects, half a million gets a second, and eleven nines of durability. Keep the facts about each object in a small replicated store, pack the bytes into large extents, and erasure-code those extents six of nine; the round is about what that costs in seeks and in repair.
The round this design follows
Why erasure coding instead of three copies?
Three copies keep 100 PB in 300 PB of disk; six data pieces and three parity pieces keep it in 150 PB and still survive any three lost pieces. At 16 TB a drive that is about 9,380 drives against 18,750. The cost is repair: rebuilding one lost piece reads six others, so a dead node takes hours to rebuild instead of under one. Both layouts clear eleven nines on the arithmetic, so the disk bill decides.
Why not cut every object into six pieces on six nodes?
Because a get would then cost six seeks instead of one. At a 1 MB median the drive time goes into seeking, and striping each object pushes the fleet to about 17,500 drives, nearly what three copies need. Packing whole objects into large extents and erasure-coding the extent keeps a normal get at one seek on one drive, and only a read that finds its piece missing reads six.
What sizes the fleet: bytes or requests?
Both, and you compute both. Uncached, 500 thousand gets and the writes of 50 thousand puts need about 18,600 drives of seeks, more than the bytes need. With four gets in five served from a cache of recent objects, seeks need about 7,140 drives and the bytes need about 9,380, so the bytes decide: 391 nodes of 24 drives. That is why the cache is in the design: it lets erasure coding pay.
Where does the metadata live, and how big is it?
In a separate store, range-partitioned by bucket and key, each partition a group of three replicas with a leader. At about 1 KB a row, ten billion objects are only 10 TB, but every get and put touches it, 650 thousand operations a second, which is 93 nodes. Metadata is small in bytes and large in operations, and that is why it is split from the data.
What happens when a data node dies?
Gets for its pieces are served by reading six other pieces of the same extent and decoding, and the placement service lists every extent that had a piece on the node. The rest of the fleet rebuilds those pieces onto nodes in other racks, each node giving up 0.2 GB/s, so with 391 nodes a dead one is rebuilt in about 8.2 hours. No object row changes, because rows point at extents and only the extent map learns the new node.