Prelude

We’ll talk a bit about how we began to incorporate an operations focus in our early history, including:

Ops by default

Unintentionally bringing in an operations focus from the beginning has influenced our engineering culture, proving to be beneficial in the future.

Learning to iterate over failure

Although we might have had the foresight with operations, we were not impervious to the common traps that startups fall into.

One of the curious aspects about the Spotify story of operations and SRE is how the initial staffing of the six-person company included an operations engineer.

Many startups add an operations-minded person only after the first customers begin using the service. The unfortunate ops engineer might then discover a backend hobbled by undocumented scripts; services running inside of screen sessions; lack of backups; single points of failure; and layers of unfinished, good intentions. From this point on, the ops engineer might constantly be playing catch-up, trying to fight fires while keeping abreast of new developments.

Including an ops engineer from the beginning, ensured that operational soundness, by extension, was included in discussions as an equal partner, not left until the last minute. From the get-go, dev and ops worked side-by-side toward our common vision of streaming music to the entire world. This initial way of working led to a culture of collaboration and trust, which continues to thrive today.

Our backend, as originally envisioned, consists of many small services—a pattern now called microservices—which together provide the content for the Spotify clients. The microservices themselves are programs that do one thing and do it well.

During these first two years, work was distributed between dev and ops as follows: developers took care of the business logic in the form of client features or backend services, whereas ops owned everything else, including detecting problems in production as well as taking care of office technology.

This was a startup, and as you might imagine, almost everything was done manually, like deployment of backend services and desktop client rollouts. As Spotify grew in the beginning, tools were added to lessen the load, but these tools only helped with manual tasks; humans still needed to make all the critical decisions.

Key Learnings

Some of our key learnings from this early period were:

• Including an ops engineer from the get-go influenced the engineering culture and proved highly beneficial as we grew.

• You should introduce an operational mindset into the engineering culture as early as possible.

• Aside from business logic architecture, ensure that an infrastructural architecture is in place, as early as possible.

Prelude

In this section, we’ll talk about how our “Ops by default” philosophy shifted and how that links up with our core engineering values:

Ops by default

Bringing in operations into regular discussions of scalability and reliability with developers from the start was pivotal to creating the foundation of our engineering approach. Operations was a cornerstone in every engineer’s work, whether it was in developing and maintaining services, or in the intrinsic urge to help improve our infrastructure.

Core engineering values

Innately trusting engineers set the foundation for one of the most prevalent principles we continue to have in the tech organization: autonomy.

Spotify went into “invite-only beta” in May 2007, and a premium (and “freemium”) option was released in 2008. During this period, Spotify experienced its first real dose of the dreaded problems that come with rapid user growth. At scale, pain points that are rare and mostly theoretical become visible. They applied not only to the backend services, but all the way down the stack. Here are a few examples:

• The backend services had capacity-related outages during peak times.

• There were disk I/O performance issues when the battery of a RAID controller failed.

• A ZFS bug caused the servers responsible for critical data to become unresponsive under high utilization.

• Racking and stacking servers was painfully slow.

Adding to the problem, the number and complexity of the backend services rose as Spotify developed new features. With time and a lot of effort, each of these technical issues was solved, and the backend continued to scale with the number of users.

However, the way in which ops engineers initially worked did not scale as well as Spotify did. There were three ops engineers at the time who owned all of the following:

• Keeping an eye on all the services that were handed over to ops from dev

• Maintaining the underlying Linux server configuration

• Responding to service disruptions

• Ensuring incident remediations were prioritized

• Keeping up with system security

• Maintaining networking equipment and configuration

• Releasing new Spotify desktop clients and monitoring for deployment failures

• Data center management including procurement, vendor relationships, and provisioning

• Maintaining and expanding storage

• Maintaining our data infrastructure and pipelines

• Owning office IT (network, firewall, staff computers and peripherals, Google apps, LDAP, Samba, Kerberos, AFS, wiki, printers, etc.)

• Serving as a support center to all of our colleagues; for example, helping with laptop setup, explaining networking configurations and how TCP/IP works, or helping our peers work with system monitoring graphs

There was quite a lot to do, but, at least initially, operations still had a little time left over for deep discussions like the future of IPv6, or bickering over the principles and pragma of the different free software licenses. However, as the different scalability issues hit us in increasingly large waves, it became clear that operations would fail spectacularly at keeping Spotify running if something wasn’t done.

Bringing Scalability and Reliability to the Forefront

Until this point, there were many people at Spotify who wanted to help with scalability and reliability-type problems, but only the ops staff was ultimately accountable for the work. But the ops team had too many responsibilities to juggle and needed to distribute those to the rest of the organization.

A weekly meeting was introduced during which all relevant backend developers and at least one ops engineer discussed services that were dealing with scalability issues. Every week, one or more services were discussed, and developer stories to scale up a system would usually trump feature work. Through this meeting, the ops engineers got some indication of which types of servers (high in disk space, CPU, or memory) was needed for the next procurement purchasing cycle.

The flavor of this meeting changed over time as dev and ops worked together. A single service’s scalability and reliability issues involved more and more dependencies on other services, so looking at each service in isolation was not enough; the focus of the meetings began to shift to the backend ecosystem as a whole.

Although we didn’t know this at the time, a transformation was taking place in our organization. Gradually, the sense of responsibility for ensuring system health was moving from being an ops-only job to something shared by all engineers working in the backend, regardless of role.

At this point in the company’s history, some of the original backend developers had root access. Seeing how useful this was, we gave root access to all developers who needed it. This level of access was unheard of in most traditional companies. Since the advent of DevOps, this is now common practice, but it wasn’t at the time. Yet it never occurred to us to do it differently; we innately trusted our engineers. The company was young, and we were united in the belief that we would take over the world of music.

By the time Spotify went out of beta, a new service would go from idea to production in the following way:

1. A developer would write and locally test the service.

2. The developer would then ask ops for one or more production servers.

3. After the servers were provisioned, the developer would log in to the host and configure the service and its dependencies.

An unintended benefit of this new flow was that developers could now understand how their services behaved in the wild better than before. The entire problem of, “Well, it worked on my computer” was bypassed. Developers could now see how their code behaved in production, view logs as they were produced, trace running processes, and even modify the operating system if necessary.

Looking back, we were unintentionally disassociating the specialized role of the SRE from the individuals in ops into a set of skills and responsibilities that could be transferred onto anyone who was capable and motivated. Over the years to come, we reused this strategy multiple times with increasing scope and depth.

And this worked—for a time.

Key Learnings

Our key learnings from this period of high growth were:

• Make operations part of the solution life cycle. Scalability, reliability, maintainability, and other operational qualities should be discussed and addressed as early as possible.

• Granting privileged access (e.g., root) to everyone who needed it removed friction, unblocking and speeding up our iteration cycles. Reliability is about achieving trust: start by trusting your own.

Prelude

In this section we’ll talk about our challenges scaling the ops team with the organization:

“Iterating over failure”

Scaling effectively continued to prove difficult, and divesting our responsibilities was not enough. We needed to revisit what operations at Spotify meant.

“Ops by default”

A central ops team doing most of the operational work doesn’t scale. We needed to make ops truly default by completely shifting the operational responsibilities closer to developers.

It was now 2012 and the Spotify user base continued to grow, introducing new scalability and stability problems for us.

The ops team was composed of fewer than 10 SREs (it was, in fact, during this year that the term “SRE” was adopted), unevenly split between Stockholm and New York. As an ops owner, each SRE was operationally responsible for dozens of backend services: handling deployments of new versions, capacity planning, system design reviews, configuration management or code reviews, and maintaining operational handbooks, among several other day-to-day operational duties.

Our backend was now running in three data centers, with a fourth on the horizon, which we had to operate and maintain. This meant that we were responsible for configuring rack switches, ordering hardware, remote hands cabling, server ingestion, host provisioning, packaging services, configuration management, and deployment—the whole chain from physical space through the application environment. As a result of this widening responsibility, we formed another team to develop tooling automation, which worked closely with ops. One of the early products developed by this team was a configuration management database (CMDB) for hardware inventory and capacity provisioning.

With tailored configurations and nonuniformity, predictability was hard. An ops owner worked closely with the dev owner of a service in efforts to improve quality, follow production readiness practices, and run through a now-formalized deployment checklist to ensure that operational standards were present even during the initial service design. Concerns we regularly brought up included the following:

• Is the service packaged and built on our build system?

• Does the service produce logs?

• Is there graphing, monitoring, and alerting?

• Are backups set up and restore tests defined?

• Was there a security review?

• Who owns this service? What are its dependencies?

• Any potential scalability concerns? Are there any single points of failure?

Manual Work Hits a Cliff

Even though we deployed constantly, either manually or via configuration management, continuous delivery was not in place. We were still delivering at a good pace, continuously, at the expense of work done by hand.

Service discovery consisted of static DNS records with manually edited and maintained zone files. DNS changes were usually reviewed and deployed by ops. We achieved mutual exclusion during DNS deploys by shouting “DNS DEPLOY!” on IRC and manually executing a script.

The ops owner and dev owner regularly went through a capacity planning spreadsheet to ensure that the service had enough capacity to sustain the current increase in usage. Access patterns and resource utilization were collected and used to predict capacity needs according to the current growth. For an ops owner of dozens of services, that meant doing a lot of capacity planning.

During the second half of 2012, we hit one million concurrent users. That meant a million people listening to music connecting directly to one of our three data centers. A pretty huge feat.

Looking back, that should be attributed to some early decisions like Spotify backend architecture being designed to scale, early client/protocol/backend optimizations, and good implementation that paid off. Also, with no multitenancy, most failures could be easily pinpointed and isolated. Every backend service did one thing and did it right. We kept it simple.

As the number of users streaming music continued to increase exponentially, the ops team couldn’t do the same. We found that the centralized ops SRE team was a system that could not scale.

Key Learnings

Our key learnings from this period were:

• Make ops truly default from the get-go; if you build it, you run and operate it.

• Shift operational responsibilities closer to the know-how: the developers.

• As your service ecosystem scales, make sure to revisit how operational work is scaling. What worked well yesterday might not work well today.

Prelude

In this section we talk about how a new approach to operations reduced bottlenecks and allowed the tech organization to grow more rapidly:

“Iterating over failure”

Adopting the new model of Ops-in-Squads freed us up to focus on minimizing our amount of manual work.

Core engineering values

Squads owning their own operations unwittingly helped us to maintain our values in autonomy and trust. Engineers had the potential to inflict widespread damage but were given the tools and processes to avoid it.

At this point, the engineering organization had become too large to operate as a single team. The Infrastructure and Operations (IO) tribe was formed, the home of teams focusing on delivering infrastructure for our backend developers and tackling the problems that came with operating at scale. One part of this tribe was called Service Availability (SA), mostly consisting of our engineers previously working in production operations. By 2013, SA consisted of four squads: security, monitoring, and two squads working on any other infrastructure tooling needed to provision and run servers. The rate at which we hired new developers and started new dev teams was too high for those four SA teams to keep up. More and more, the ability to deliver new or improved features slowed down due to the pace at which we could buy and rack new servers, review and merge Puppet changes, add DNS records, or set up alerting for a service. Dozens of feature teams were impatiently waiting to get their changes out to our users but had to wait on us for code reviews or provision hardware for their services.

We were also still on call for core services, and the operational quality and stability of those services suffered as we tried to keep up with our accumulation of tasks. The global “operations backlog” just kept on growing and growing.

Building on Trust

However, as we removed these “sanity checks” previously done by the ops team, we faced a new problem. Without the guidance of the ops team, the entropy in our backend exploded. Poorly tested services and simple experimentations made it into the production environment, sometimes causing outages and paging our ops on-call engineers who now lacked the required understanding of what was running in production. We needed to figure out how to get the knowledge and operational responsibilities back into one place, paging the right people to troubleshoot an incident.

The traditional ops-owner approach worked well when Spotify was a small organization, but it had complexity and scalability issues, both technical and organizational. It became impossible to rely on a single or a pair of systems owners to figure them all out. We also realized that acting as a gatekeeper to operations meant we kept a monopoly on operational learning opportunities.

Handing this responsibility over to the backend teams seemed like the logical next step. The next question was how to proceed. We needed to actively engage the teams and figure out a way forward with the rest of the tech organization.

We started rolling out a new way of working—dubbed Ops-in-Squads—which in essence included everything needed to hand over on-call and operational responsibility for services to the teams that developed them. We needed to write tooling to do many of the things that were done by hand before; we needed better documentation and training so that developers would be able to troubleshoot production issues; and we needed buy-in from developers to drive this change.

A guild created for developers and ops-people to share knowledge among everyone and open a bidirectional communication channel helped define some key things we needed:

• A standardized way of defining Service-Level Agreement (SLAs; we did not use the Service-Level Indicator [SLI]/Service-Level Objectives [SLO] concepts back then)

• A how-to crash course on being on call, handling incidents, performing root-cause analysis, and holding postmortems

• Guidance on capacity planning

• Best practices for setting up monitoring and alerting as well as instruction on interpreting monitoring data

• Training in troubleshooting, system interactions, and infrastructure tooling

Handing all of this responsibility over to the teams would of course be a big undertaking over a long period of time, and the infrastructure team still remained to support and build upon the core platform. We struggled with defining exactly what fell into the core platform; some things were obvious, like networking, monitoring, and provisioning, but other systems were harder to classify. Critical parts of our infrastructure, like the user login service, the playlist system, or the song encryption key system, needed to be evaluated: are they core infrastructure, or should they be handled like any other backend system?

We wanted to make a deal: we remove gatekeeping and friction points in exchange for shifting operational responsibilities into feature teams. If a team needs to deploy a change on a Friday, it shouldn’t be prevented from doing so. We assumed that the knowledge for operating services is held closest to where it is developed.

This was not met with cheers and optimism from everyone in the organization; there were concerns about negatively impacting feature teams’ speed of iteration. How would teams have sufficient bandwidth to deliver on features and growth if they also handled the maintenance and operational duties of their systems? The teams consisted of developers, not SREs; how much of their time would they have to devote to learning and practicing operations? Preparation, process, and tooling would be essential to convince people to take on this responsibility.

We continued investing heavily in picking blessed tooling, writing and promoting frameworks, and writing documentation to help make the transition as smooth as possible for most teams.

Driving the Paradigm Shift

We aim to make mistakes faster than anyone else.

Daniel Ek, Spotify founder and CEO

To reach all the teams with the information and tools needed, we took several different approaches. We held presentations for developers, discussing how the systems in our backend fit together and how to use the shared infrastructure. Our principal architect internally held postmortems for dozens of developers, walking them through a large incident that had resulted from a cascading failure. We made it easier for teams to find documentation and ownership for infrastructure systems by having a central entry point. We developed an Ops-in-Squads handbook that became the standard document to which to point new developers. It contained most of the information that teams needed to get started; pointers on things to do; where to read more; and checklists of processes, technical features, or processes to implement. We embedded in backend teams for short periods to transition the host team to being on-call for their services; helped teach operations; and worked directly with improving the systems, deployment procedures, on call handbooks, and so on for teams that requested or needed it. Again, we made a deal: we’ll work with the team for a few sprints and “clean house” before handing over the on-call responsibilities to them.

In 2014, this approach—embedding engineers in teams—was expanded into an Ops-in-Squads tour where some parts of our operations team went week by week to different teams, engaging with their daily work and trying to help them on anything with which they struggled.

During these embeds, we reviewed the architecture of the host teams and looked at alerting and monitoring, helping to improve these when needed. We also discussed how to do on-call and shared best practices around schedule rotation, how to escalate problems, and how to hold postmortems. Postmortems, in particular, had a reputation of being time-sink ceremonies. Teams that had a lot of incidents often skipped postmortems because the work to establish timelines, find root causes, and define remediations for 10 or 20 incidents per week seemed like too much overhead. Finding different ways of approaching postmortems proved useful in these cases, like clustering incidents after a theme, or doing short postmortems for multiple incidents at once. Often many incidents had similar root causes; exact timelines weren’t as important as identifying the top remediations that would reduce the likelihood of that type of incident by 90% in the future. Throughout this shift, we retained the principle of blameless postmortems, emphasizing the importance of learning from our mistakes, and ensuring that no one was thrown under the bus.

Key Learnings

Some of our key learnings from this period were:

• Strive to automate everything. Removing manual steps, friction, and blockers improves your ability to iterate on product.

• Ensure that self-service operational tooling has adequate protection and safety nets in place.

• Teaching ops through collaboration is vital to making ops “default” in the organization.

Prelude

In this section we’ll talk a bit about how we tried to balance autonomy for squads with consistency in the tech stack:

Iterating over failure

Our first approach at introducing consistency in the technology stack caused unintentional further fragmentation. Although we continued to reiterate and address these newly exposed concerns, we found ourselves blocking teams once again.

Core engineering values

Focusing on unblocking feature teams allowed us to maintain squad independence and freedom while introducing much-needed infrastructure consistency.

In moving forward with the Ops-in-Squads model, we shifted our focus to standardizing our technology stack. With this decentralized operations model, we needed to reduce the cost of operations for teams by providing consistency across our infrastructure. High entropy meant expensive context switches and unnecessary overhead. However, striking a balance between consistency and our penchant for autonomy took thoughtfulness and foresight.

Bringing uniformity to Spotify’s stack came in a few forms. Although teams were now in charge of operating their services, we needed to get out of their way by building abstraction layers in the right places. The first levels of abstraction layers were tools for ourselves, iterating on our early successes of removing friction points with provgun and DNS. Out of this work came a number of tools: moob, for out-of-band management of hardware; neep, a job dispatcher to install, recycle, and restart hardware; and zonextgen, a batch job to create DNS records for all servers in use, to name a few.

Building off of the earlier legwork, in 2015 through 2016 we concentrated on creating and iterating on self-service tooling and products for feature developers around capacity management, Dockerized deployments, monitoring, and SLA definitions. No longer did developers need to plan capacity with spreadsheets or deploy via SSH’ing into servers to do an apt-get install. A few simple clicks of a button and squads got the capacity they needed with their service deployed.

We established a blessed stack in 2015, one that is explicitly supported and maintained by the infrastructure team, that became the “Golden Path” for how a backend service is developed, deployed, and monitored at Spotify. We began using the term Golden Path to describe a series of steps that was supported, maintained, and optimized by our infrastructure team. Instead of dictating a “blessed stack” as a mandatory solution, we wanted to make the Golden Path so easy to use that there was almost no reason to use anything else.

The Golden Path consisted of step-by-step guide for our developers. It included how to set up their environment; create a simple, Dockerized service; manage secrets; add storage; prepare for on-call; and, finally, properly deprecate a service. We built Apollo, our Java microservice framework, which would give a developer many features for free, like metrics instrumentation, logging, and service discovery. Heroic and Alien, our time series database (TSDB) and frontend, allowed engineers to create prepackaged dashboards and alerts of their instrumented-by-default Apollo services. Pairing with Helios then provided a supported way to roll out Docker deployments in a controlled manner with zero downtime.

Benefits

Squads were now able to focus even more on building features because operational tasks were continuously being minimized. But developer speed was not the only benefit to improving the consistency within our infrastructure. It allowed our migration from our physical data centers to Google Cloud Platform to be fairly seamless. Most of the work happened behind the scenes from the developer point of view. Creating capacity in Google Cloud Platform was no different than on bare metal. Neither was deploying or monitoring a service. From the developer point of view, moving to a new squad or doing a temporary embed became frictionless, given a consistent set of tools to work with.

Another benefit to a consistent, blessed stack is that it prevents us from falling into potential traps of trendy technologies that come and go. We also have insights and understanding about what engineers are using, informing us how to improve our products. There is less operational complexity and fragmentation, which can be a nightmare during incidents. We’re able to prescribe best practices for operating a service, and provide ongoing support. As an infrastructure team, we decided to have fewer responsibilities, but strived to handle them well.

Trade-Offs

It was not all easy sailing, though. Standardizing a technology stack still had its drawbacks. Too restrictive, and we risk losing squad autonomy and experimentation, hindering development when not all use cases are addressed. We defined the Golden Path to be an Apollo service, but hadn’t yet provided explicit support for the legacy and internal Python services, frontend applications, or data pipelines and analytics. We provided the ability to easily create and destroy capacity, but we didn’t yet generally support autoscaling. These unaddressed use cases indirectly contributed to the fragmentation that we were confronting as some teams created their own bespoke tools for workarounds.

In some cases, supported tools were not used. JetBrain’s TeamCity used to be the only Continuous Integration (CI) system used. It became too cumbersome for backend service developers, so squads spun up their own Jenkins instances. As evidence of team autonomy and experimentation, the use of Jenkins spread so quickly among squads that it soon became the de facto backend service CI tool. It made sense with the Ops-in-Squads model we adopted, except that teams lacked good habits to maintain their own Jenkins setup, leaving them out of date, vulnerable, and inadequately secured. It forced us to rethink our build pipeline support, where we ultimately developed an explicitly supported managed Jenkins service. Similarly, we found developers were not maintaining their Cassandra clusters, despite tools provided by our team. Fearing potential data loss and realizing maintenance was too much overhead for a feature team, we brought this, too, back into our operational ownership by offering a managed Cassandra service and related support.

Even though we had many self-service tools to unblock feature teams, the IO tribe still got in the way. In 2017, our focus shifted again to strengthening our overall infrastructure by prioritizing ephemerality, security, and reliability. We built tools for initiating controlled rolling reboots of our entire fleet, which encouraged developers to write resilient services. We automated regional failover exercises, bringing to light inconsistencies that teams have in their services’ capacity across regions. As we make progress in taking advantage of products and services that Google Cloud Platform offers, teams are now needing to take time away from feature development to, for instance, migrate to a new cloud-native storage solution such as Bigtable but reap the benefits when they no longer need to maintain any infrastructure.

Although we succeeded in shifting to the Ops-in-Squads model and struck a balance between autonomy and consistency, we are now focusing on removing friction from operations—but we still have a long road ahead.