Reliability, Scalability, Maintainability: the three pillars
The vocabulary for judging any data system: does it keep working, can it grow, and can humans live with it?
Retrieve before reading
In one sentence: your service is down because a disk died. Your boss asks "why didn't we plan for that?" What is the honest answer?
Reveal a strong answer
Something like: "We planned for disks dying - we have RAID and backups. The disk was not the problem: the software assumed a disk read can never return garbage, and that assumption broke. Hardware faults are expected; the failure was a software error that turned a single fault into an outage."
What the three pillars are
Most data systems are judged on three concerns, and the whole book is organized around them. Reliability: the system continues to work correctly even when things go wrong. Scalability: as the system grows, there are reasonable ways to cope. Maintainability: over time, many different people can work on it productively. These words get thrown around without meaning; this lesson gives each one a precise handle.
Reliability: faults, not failures
The things that go wrong are called faults, and a system that copes with them is fault-tolerant or resilient. The key distinction: a fault is one component deviating from its spec; a failure is the system as a whole stopping to serve the user. You cannot reduce the probability of faults to zero, so the design goal is to stop faults from becoming failures.
There is a counter-intuitive practice that follows from this: deliberately increase the fault rate. Netflix's Chaos Monkey randomly kills processes in production, on purpose. The reasoning: most critical bugs are poor error handling, and the only way to keep the fault-tolerance machinery exercised is to trigger it constantly. A fault-tolerance path that has never run is a bug waiting to happen.
Hardware faults
Disks crash, RAM goes bad, power fails, someone unplugs the wrong cable. With enough machines these are routine: a disk has a mean time to failure (MTTF) of roughly 10-50 years, so a cluster of 10,000 disks loses about one disk per day on average. The classic response is redundancy - RAID, dual power supplies, backup generators. That still works for a single machine, but as systems spread across more machines, the fault rate rises proportionally, and cloud platforms treat single-machine loss as normal. The modern move is to tolerate the loss of whole machines in software, which has a side benefit: you can patch one node at a time without ever taking the system down.
Software errors
Hardware faults are roughly random and independent. Software faults are systematic: the same bug hits every instance at once, which is why they cause more outages than hardware. The classic example: the leap second on June 30, 2012, which made many Linux applications hang simultaneously. A software fault often hides as an assumption about the environment - "the clock never jumps backwards", "this input is always well-formed" - that is true for years and then stops being true. There is no quick fix; the toolkit is careful thinking about assumptions, thorough testing, process isolation, crash-and-restart, and constant self-checking in production.
Human errors
Operators are human, and configuration errors by operators are the leading cause of outages in large internet services - hardware faults play a role in only 10-25% of outages. The best systems combine several defenses: design interfaces that make the right thing easy and the wrong thing hard; decouple the places where people make mistakes from the places where they can cause damage (sandboxes with real data); test at all levels; make recovery fast (roll back configs quickly, roll out code gradually); and monitor with clear telemetry.
Reliability is not only for nuclear power stations. A photo app that loses a parent's pictures of their children is a reliability failure with a human cost. Choosing to cut reliability corners for a prototype is a legitimate business decision - but it should be a conscious one, not an accident.
Scalability: load parameters and percentiles
"X is scalable" is meaningless on its own. Scalability is a question: if the system grows in a particular way, what are our options for coping? The first step is to describe the load with a few numbers - the load parameters. The right choice depends on the architecture: requests per second for a web server, read/write ratio for a database, active users in a chat room, cache hit rate. Sometimes the average matters; sometimes the bottleneck is a small number of extreme cases.
The canonical example: Twitter's fan-out
The book's worked example (November 2012 numbers) is worth understanding in full, because it shows how a load parameter can be the whole story. Two operations: post tweet (4.6k requests/sec average, 12k/sec peak) and home timeline (300k requests/sec). The naive implementation - store tweets in one table, and on every timeline read, join tweets with follows and merge - could not keep up. The fix: fan-out. When a user posts, write the tweet into each follower's timeline cache ahead of time. Reads become cheap; writes become expensive: 4.6k tweets/sec times an average of 75 followers is 345k writes/sec to the caches. And the average hides the tail: some users have over 30 million followers, so a single tweet can mean 30 million writes.
The final twist: Twitter ended up with a hybrid. Most users' tweets are fanned out at post time, but a small number of celebrities are exempted - their tweets are fetched and merged at read time instead. The lesson generalizes: the load parameter that matters is the one that determines your fan-out, and the architecture is a deliberate trade between write-time work and read-time work.
Describing performance: percentiles, not averages
Response time is not a single number; it is a distribution. The mean is a poor summary - it tells you nothing about how many users actually experienced the delay. The standard tool is percentiles: sort all response times, and the p50 (median) is the halfway point, p95 the threshold below which 95% of requests fall, p99 below which 99% fall. High percentiles matter because they hit real users: Amazon tracks the 99.9th percentile for internal services, because the people who get the slowest responses tend to be the biggest spenders - and Amazon observed that a 100 ms increase in response time reduced sales by 1%.
Two practical consequences. First, head-of-line blocking: a server processes a limited number of requests in parallel, so a few slow requests hold up everything behind them - measure on the client side, because that is what users see. Second, when one end-user request fans out into many backend calls, it takes just one slow backend call to make the whole request slow; the chance of at least one slow call grows with the number of calls. This is why p99 of a single service is not the p99 of the user experience.
Coping with load
An architecture that works at one load level will not survive ten times that load; expect to rethink the architecture at every order of magnitude. The classic dichotomy: scaling up (a more powerful machine) vs scaling out (more machines, shared-nothing). Real systems are a pragmatic mixture. Some systems are elastic - they add resources automatically when load rises; others are scaled manually, which is simpler and has fewer operational surprises. And there is no magic scaling sauce: a system built for 100,000 requests/sec of 1 kB each looks nothing like one built for three requests/min of 2 GB each, even though the throughput is the same. For an early-stage product, iterating on features usually beats scaling for hypothetical future load.
Maintainability: the cost nobody budgets for
The majority of the cost of software is not the initial build; it is the ongoing maintenance - fixing bugs, keeping systems operational, adapting to new use cases, repaying technical debt. Three design principles keep maintenance pain low:
- Operability - keep routine operational toil low: monitoring, rollbacks, and day-to-day running should be easy for the operations team. The book's framing: a skilled ops team can prop up mediocre software, but no amount of good software survives careless operations.
- Simplicity - keep the system easy for a new engineer to grasp by removing accidental complexity (complexity that is not inherent to the problem). This is not about the user interface; it is about the code and architecture.
- Evolvability - keep the system easy to adapt when requirements change in ways you did not anticipate. Also called extensibility or plasticity.
Every data system is a bundle of trade-offs. The three pillars give you the vocabulary to name the trade-off you are making - and the book's later chapters give you the mechanisms to make it deliberately.
Problem set
P1. Fault or failure?
Classify each event as a fault, a failure, or both, and justify: (a) one database replica in a three-replica cluster crashes; (b) the cluster stops answering reads because the two remaining replicas disagree; (c) an operator typo in a config file takes down the whole service.
Solution
(a) Fault: one component deviates from spec; the system as a whole still serves. (b) Both: the replica disagreement is itself a fault (an invariant violation), and the cluster's inability to answer reads is the resulting failure. Do not assume the original crash caused the disagreement unless that is stated - the disagreement could have its own cause. (c) Both: the typo is a fault (human error), and the outage is a failure. The interesting question is why a single typo could take down the whole service - that is a design failure in the fault-tolerance machinery.
P2. Compute fan-out
A social app has 2,000 posts/sec on average. The average user has 40 followers, but 0.1% of users have 1,000,000 followers. (a) What is the average fan-out write rate if every post is delivered to every follower's inbox at post time? (b) Roughly how many writes per second do the celebrity posts alone add, if celebrities post at the same average rate as everyone else?
Solution
(a) \(2{,}000\) posts/sec \(\times 40\) followers = \(80{,}000\) writes/sec on average.
(b) Celebrity posts: \(0.1\%\) of \(2{,}000\) = \(2\) posts/sec, each fanned out to \(1{,}000{,}000\) followers = \(2{,}000{,}000\) writes/sec - 25 times the entire average fan-out. This is the Twitter lesson: the average hides the tail, and the tail is where the architecture breaks. The hybrid fix (exempt celebrities, merge at read time) exists precisely because of this arithmetic.
P3. Percentiles from a sample
Ten requests have response times (ms): 80, 95, 100, 110, 120, 130, 150, 200, 400, 900. Compute the mean, p50, p95, and p99. What does each number say about the user experience?
Solution
Mean = \((80+95+100+110+120+130+150+200+400+900)/10 = 2285/10 = 228.5\) ms.
Sorted, p50 = \((120+130)/2 = 125\) ms (median of 10 values). p95: with 10 samples, the 95th percentile is between the 9th and 10th values; conventionally the 9.5th position, i.e. \((400+900)/2 = 650\) ms. p99 is beyond the largest sample, so it is estimated at 900 ms or higher - with 10 samples you cannot actually estimate p99.
The mean (228.5 ms) is dragged up by the two slow requests and says "fine"; the median says "half of users wait under 125 ms"; p95 says "5% of users wait 650 ms or more". The 900 ms outlier is invisible in the mean but dominates the tail - and if this service is called 10 times per page load, the chance a user hits the 900 ms call is far higher than 5%.
P4. The slowest call wins
A page load makes 5 backend calls in parallel. Each call has a 2% chance of taking 2 seconds (the rest take 50 ms). (a) What fraction of page loads take 2 seconds? (b) What if the calls are sequential instead of parallel?
Solution
(a) The page waits for the slowest call. \(P(\text{no call slow}) = 0.98^5 \approx 0.9039\), so about 9.6% of page loads take 2 seconds - even though each individual call is slow only 2% of the time. This is the book's point about fan-out into backend calls: the tail of the user experience is much worse than the tail of any single call.
(b) Sequential: expected time = \(5 \times (0.98 \times 50 + 0.02 \times 2000) = 5 \times 89 = 445\) ms, and the 2-second case only happens when a slow call occurs (about 9.6% of loads, same probability, but the cost is \(2000 + 4 \times 50 \approx 2.2\) s; with \(k\) slow calls the total is \(2000k + 50(5-k)\) ms). Parallel: the page waits for the slowest call, so the expected time is \(50 \times 0.98^5 + 2000 \times (1 - 0.98^5) \approx 237\) ms - better than 445 ms on average, but the tail probability is the same 9.6%, and the tail is dominated by the slowest call either way. Note 89 ms is only the mean of one call, not of the page load.
P5. Design the load parameters
You are designing a food delivery app. Name the load parameters you would track for: (a) the restaurant-facing order API, (b) the customer-facing menu browsing API, (c) the courier location update stream. For each, say whether the average or the tail is the better design driver, and why.
Solution
(a) Orders: writes/sec and peak-hour burst rate; the tail matters because a rejected order is a lost sale - capacity must cover the peak, not the average.
(b) Menu browsing: reads/sec and cache hit rate; the average matters more, but p99 still drives perceived speed - a slow menu page loses customers.
(c) Couriers: writes/sec per courier × number of couriers (a fan-out-style product: total write rate scales with courier count), plus the per-location update frequency. The average matters for capacity planning, but the peak (dinner rush) is the design driver.
The general rule from the lesson: the load parameter is the one that determines the architecture's bottleneck - for Twitter it was fan-out, for a delivery app it might be peak-hour write bursts.
P6. Interpret the artifact
Pick a service you use or run (a chat app, a CI system, a database, anything with users). In plain language, as if to a non-engineer: (a) what would "reliable" mean for it, and which fault class (hardware, software, human) is most likely to break it; (b) what is its most important load parameter, and what happens when it doubles; (c) one thing that makes it hard to maintain, and which of the three principles (operability, simplicity, evolvability) would fix it.
Solution
A strong answer names concrete things, not platitudes. Example for a chat app: (a) reliable = messages arrive once, in order, and the app stays usable during a server outage; the most likely fault class is software (a bug in the message-ordering logic) because the hardware is redundant but the ordering logic is not. (b) Load parameter = messages/sec and active connections; when it doubles, the message queue becomes the bottleneck and delivery latency grows. (c) Hard to maintain because the message-ordering logic is tangled with the UI code; evolvability is the fix - separate the ordering core from the presentation so future changes don't risk the ordering guarantees.
The point of the exercise: the three pillars are not abstract - you should be able to name the fault, the load parameter, and the maintenance pain for any system you touch.