Restoration

Restoring state from backups can take a surprisingly long time, especially if you haven’t practiced recovering from backup recently. In fact, if you haven’t tested your backups recently, it’s possible they don’t even work at all!

In the early days of Dropbox, we were dismayed to discover that it was going to take eight hours just to restart a production database after a catastrophic failure. Even though all of the data was intact, we had set the innodb_max_dirty_pages_pct MySQL parameter too high, resulting in MySQL taking eight hours just to scan the redo log during crash recovery.¹ Fortunately, this database didn’t store critical data, and we were able bring dropbox.com back online in a couple of hours by bypassing the database altogether, but this was certainly a wake-up call. Our operational sophistication has improved dramatically since then, which I outline later in this chapter.

Freshness

Backups represent a previous snapshot in time and will usually result in loss of recent data when you restore from them. You typically want to store both full-snapshot backups for fast recovery and for historical versioning, along with incremental backups of more recent state to minimize staleness. Any stronger guarantees than this, however, require an actual replication protocol.

Estimating durability

Regardless of the choice of replication technique, an obvious question you might have is, “How much durability do I actually have?” Do you need one database replica? Two? More?

One convenient way of estimating durability is to plug some numbers into a Markov model. Let’s jump into some (simplified) math here, so take a deep breath—or just skip to the end of this section.

Let’s assume that each disk has a given mean time to failure (MTTF) measured in hours and that we have operational processes in place to replace and re-replicate a failed disk with a given mean time to recovery (MTTR) in hours. We’ll represent these as failure and recovery rates of $\lambda =\frac{1}{\mathit{\text{MTTF}}}$ and $\mu =\frac{1}{\mathit{\text{MTTR}}}$, respectively. This means that we assume that each disk fails at a rate of λ failures per hour on average, and that each individual disk failure is replaced at a rate of μ recoveries per hour.

For a given replication scheme, let’s say that we have n disks in our replication group, and that we lose data if we lose more than m disks; for example, for three-way database replication we have n = 3 and m = 2; for RS(9,6) erasure coding, we have n = 9 and m = 3.

We can take these variables and model a Markov chain, shown in Figure 17-1, in which each state represents a given number of failures in a group and the transitions between states indicate the rate of disks failing or being recovered.

Figure 17-1. Durability Markov model

In this model, the flow from the first to second state is equal to nλ because there are n disks remaining that each fail at a rate of λ failures per hour. The flow from the second state back to the first is equal to μ because we have only one failed disk, which is recovered at a rate of μ recoveries per hour, and so on.

The rate of data loss in this model is equivalent to the rate of moving from the second-last state to the data-loss state. This flow can be computed as

for a data loss rate of R(loss) replication groups per hour.

This simplified failure model allows us to plug in some numbers and estimate how likely we are to lose data.

Let’s say our disks have an Annualized Failure Rate (AFR) of 3%. We can compute MTTF from AFR by using $\mathit{\text{MTTF}}=\frac{-8,766}{\ln (1-\mathit{\text{AFR}})}$ (this is approximately $\frac{8,766}{\mathit{\text{AFR}}}$); in this case MTTF = 287,795 hours. Let’s say we have some reasonably good operational tooling that can replace and re-replicate a disk within 24 hours after failure, so MTTR = 24. If we adopt three-way data replication we have n = 3, m = 2, $\lambda =\frac{1}{287,795}$, and $\mu =\frac{1}{24}$.

Plugging these into the equation earlier, we get R(loss) = 7.25 × 10^–14 data loss incidents per hour or 6.35 × 10^–10 incidents per year. This means that a given replication group is safe in a given year with probability 0.9999999994.

Wait, that’s pretty safe, right?

Well, nine 9s of durability is pretty decent. If you have a huge number of replication groups, however, failure becomes more likely than this, because each individual group has nine 9s of durability, but in aggregate it’s more likely that one will fail. If you’re a big consumer storage company, you’ll likely want to push durability further than this. If you reduce recovery delay or buy better disks or increase replication factor, it’s pretty easy to push durability numbers much higher. Dropbox achieves theoretical durability numbers well beyond twenty-four 9s, aided by some fancy erasure coding and automatic disk recovery systems. According to these models, it’s practically impossible to lose data. Does that mean we get to pat ourselves on the back and go home?

Unfortunately not…

Your durability estimate is only an upper bound!

It’s fairly easy to design a system with astronomically high durability numbers. Twenty-four 9s is an MTTF of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe, it might be time to reevaluate your priorities.

Should we trust these numbers, though? Of course not, because the secret truth is that adherence to theoretical durability estimates is missing the point. They tell you how likely you are to lose data due to routine disk failure, but routine disk failure is easy to model for and protect against. If you lose data due to routine disk failure, you’re probably doing something wrong.

What’s the most likely way you’re going to lose data? Is it losing a specific set of half a dozen different disks across multiple geographical regions within a narrow window of time? Or, is it an operator accidentally running a script that deletes all your files, or a firmware bug that causes half your storage nodes to fail simultaneously, or a software bug that silently corrupts 1% of your objects? Protecting against these threats is where the real SRE work comes in.

Physical isolation

To put it succinctly, store your stuff on different stuff.

There’s a wide spectrum of physical failure domains that spans from disks to machines, to racks, to rows, to power feeds, to network clusters, to data centers, to regions or even countries. Isolation improves as you go up the stack, but usually comes with a significant cost, complexity, or performance penalty. Storing state across multiple geographic regions often comes with high network and hardware costs, and often entails a big increase in latency. At Dropbox, we design data placement algorithms taking into consideration the power distribution schematics in the data center to minimize the impact of a power outage. For many companies this is clearly overkill.

Every company needs to pick an isolation level that matches its desired guarantees and technical feasibility, but after a level is chosen, it’s important to own it. “If this data center burns down, we’re going to lose all of our data and the company is over” is not necessarily an irresponsible statement, but it is irresponsible to be caught by surprise when you’re unsure of what isolation guarantees you actually have.

There are other dimensions to physical isolation, like dual-sourcing your hardware or operating multiple hardware revisions of disks and storage devices. Although this is outside the scope of many companies, large infrastructure providers will usually buy hardware components from multiple suppliers to minimize the impact of a production shortage or a hardware bug. You’d likely be surprised how often bugs show up in components like disk drivers, routing firmware, and RAID cards. Judicious adoption of hardware diversity can reduce the impact of catastrophic failures and provide a backup option if a new hardware class doesn’t pass production validation testing.

Logical isolation

Failures tend to cascade and bugs tend to propagate. If one system goes down, it will often hose everything else in the process. If one bad node begins writing corrupt data to another node, that data can propagate through the system and corrupt an increasingly large share of your storage. These failures happen because distributed systems often have strong logical dependencies between components.

The key to logical isolation is to build loosely coupled systems.

Logical isolation is difficult to achieve. Database replicas will happily store corrupt data issued to them by the primary database. A quorum consensus protocol like Zookeeper will also suffer the same fate. These systems are not only tightly coupled from a durability perspective, they’re tightly coupled from an availability perspective: if a load spike leads to a failure of one component, there is usually an even greater load applied to the other components, which subsequently also fail.

Strong logical isolation usually needs to be designed into the underlying architecture of a system.

One example at Dropbox is region isolation within the storage system. File objects at Dropbox are replicated extensively within a single geographical region, but also duplicated in a completely separate geographical region on the other side of the country. The replication protocols within a region are quite complicated and designed for high storage efficiency, but the API between the two regions is extremely simple: primarily just put and get, as illustrated in Figure 17-2.

Figure 17-2. Isolated multiregion storage architecture

Strong isolation between regions is a nice way to use modularity to hide complexity, but it comes at a significant cost in terms of replication overhead. So why would a company spend more than it needs to for storage?

The abstraction boundary between storage regions makes it extremely hard for cascading issues to propagate across regions. The loose logical coupling and simple API makes it difficult for a bug or inconsistency in one region to impact the other. The loose coupling also makes it possible to take an entire region down in an emergency without impacting our end users; in fact, we take a region down every week as a test exercise. This architecture results in a system that is extremely reliable from an availability and durability perspective, without imposing a significant operational burden on the engineers who run the system.

Operational isolation

The most important dimension to isolation is operational isolation, yet it is one that is often overlooked. You can build one of the world’s most sophisticated distributed storage systems, with extensive replication and physical and logical isolation, but then let someone run a fleet-wide firmware upgrade that causes disk writes to fail, or to accidentally reboot all your machines. These aren’t academic examples; we’ve seen both happen in the early days of Dropbox.

Typically, the most dangerous component of a system is the people who operate it. Mature SRE organizations recognize this and build systems to protect against themselves. I elaborate more on protections in the next section, but one critical set of protections is isolation across release process, tooling, and access controls. This means implementing restrictions to prevent potentially dangerous batch processes from running across multiple isolation zones simultaneously.

Operational isolation is enforced at many layers of the Dropbox stack, but one particular example is isolation in the code release process for the storage system. Some parts of the Dropbox code base are pushed to production every day, in use cases for which correctness or durability aren’t at stake, but for the underlying storage system, we adopt a multiweek release process that ensures that at least one fully replicated copy of data is stored on a thoroughly vetted version of the code at all times, as depicted in Figure 17-3.

Figure 17-3. Storage system code release process.

All new versions of storage code go through an extensive unit testing and integration testing process to catch any trivial errors, followed by overnight end-to-end durability testing on a randomized synthetic workload. We also run a set of long-running version-skew tests during which different code versions are run simultaneously on different nodes in the system; code pushes aren’t atomic, so we need to ensure that old and new versions of code are compatible with each other. After these tests have all passed, the release is automatically marked as “green” and is ready to be pushed to the staging cluster.

The staging cluster is an actual production deployment of the storage system that stores a mirror of a subset of production data. This is a relatively large cluster storing many tens of petabytes across multiple geographical regions. The code runs on this cluster for a full week and is subjected to the same workloads, monitoring, and verification as the rest of the storage system. After a week in staging the individual “DRIs” (directly responsible individuals—the engineers who “own” each subsystem) sign off that each software component is operating correctly and without any significant performance degradation. At this point, the software is ready to be pushed to the first real production region.

Code is pushed to the first production region and runs for another week before it can be safely pushed to the remaining regions. Recall that the multiregion architecture of the storage system ensures that all data in one region is replicated in at least one other region, avoiding any risk of data loss from a single code push. Internal verification systems are tuned to detect any anomalies within this one-week timeframe, but in practice, any potential durability issues are caught well before making it this far. The multiregion deployment model is tremendously valuable, however, at catching any performance or availability issues that arise only at large scale.

The release process is pipelined so that there are always multiple versions of code rolling out to production. The process itself is also automated to a large extent to avoid any operator errors. For practical reasons, we also support a “break-glass” procedure to allow sign-off on any emergency changes that need to be fast-tracked to production, along with extensive logging.

The thoroughness in this release process allows us to provide an extremely high level of durability, but it’s clearly overkill for the vast majority of use cases. Operational isolation comes at a cost, which is usually measured in inconvenience or engineer frustration. At a small company, it can be incredibly convenient to run a batch job across all machines simultaneously, or for engineers to be able to SSH into any server in the fleet. As a company grows, though, investments in operational isolation not only guard against major disasters, but also allow a team to move faster by providing guardrails within which to iterate and develop quickly.