Design a metrics and alerting system
Ten million series, a fresh point for each every ten seconds, thirteen months kept, and a page within thirty seconds. The round turns on two choices: pull the samples so a silent host is itself a signal, and pack them into columns so a sample costs a byte and a half instead of a row.
The round this design follows
Should a metrics system pull from targets or have them push?
Pull, for the fleet, with a push gateway for batch jobs. When the monitoring side scrapes every target on a ten-second tick, a target that stops answering is noticed on the next tick, because the scrape itself fails; with push, a dead host and a quiet host look the same. Pull also puts the rate in the monitoring system's hands, so one misbehaving client cannot flood ingest. The cost is service discovery, and a job that ends between two scrapes, which is what the push gateway is for.
How much data is ten million series scraped every ten seconds?
A million samples a second. Stored as rows of series id, time and value at about 64 bytes each, thirteen months would be 2.16 PB. Packed into compressed chunks at about 1.5 bytes a sample, with the last 30 days at full resolution and the rest downsampled to five-minute points, it is 12.3 TB. Say both numbers out loud: the gap between them is the reason to use a time-series store rather than a general database.
Why does a time-series database compress so well?
Timestamps arrive on a fixed tick, so the gap between two samples is almost always the same and the change in the gap is zero, which packs into one bit. Values change slowly, so the XOR of a value with the previous one is mostly zero bits, and only the few bits in the middle are stored. Together that is delta-of-delta encoding for time and XOR encoding for values, and it takes a 16-byte sample down to about a byte and a half.
What is a cardinality explosion, and how do you stop one?
Every distinct set of label values is its own series, so a label with many values multiplies series. A customer id label with a hundred thousand values on a metric that had a hundred series creates ten million new series, which doubles the fleet's memory. Hashing spreads those series across ingesters but cannot shrink them. The fix is a limit at ingest: a cap on series per metric and per scrape, refusals counted in a metric of their own, and per-customer data moved to a store built for it.
How fast can an alert fire?
Add up the worst case of each stage. A problem can start just after a scrape, so the sample shows up to ten seconds later; the rule is evaluated every ten seconds; the notifier waits five seconds to group alerts. That is 25 seconds, inside a 30-second budget. A for duration on the rule, which makes a condition hold for a while before it fires, is added on top on purpose, to keep one bad scrape from paging anyone.