Experience is something you don’t get until just after you need it.
—Steven Wright
Management shouldn’t inflict sameness.
—Jonathan Smart
If a man is called to be a street sweeper, he should sweep streets even as Michelangelo painted, or Beethoven composed music, or Shakespeare wrote poetry. He should sweep streets so well that all the hosts of heaven and earth will pause to say, here lived a great street sweeper who did his job well.
—Martin Luther King Jr.
If you were to ask anyone on Ben’s team how things are actually going, I imagine you’d get mixed responses. Some would be enthusiastic. Others would point to some nasty skeletons in the closet that remain. In other words, work is still work, and it doesn’t rain puppies, kittens, and rainbows now that they’ve discovered this magical word “DevOps.”
Still, most on the team would agree that things are getting better, that they’re climbing out of a rut – albeit with a long way to go. Let’s take a look back from a year ago and see how the team has grown and the challenges they’ve faced:
That’s a gigantic list when you stare at it, even if their progress isn’t perfect on any of these points. Gradually, with fits and starts, an ecosystem is taking shape that’s more learning-friendly and robust, resilient, able to cope with disaster and broken releases better.
With Kanban , they’re limiting WIP, meaning they’re causing pain by forcing the team to focus on driving one or two things forward to “done” at a time. This exposes problems with lack of automation, handoffs, bad tooling, signoff hassles, etc. It also means the team is able to work more effectively (less loss due to context switching) and demonstrate what they are working on in dashboards – transparency. Without it, the team is not really a team but a bunch of disconnected siloes, and blockers are not really being exposed as they should be.
With Version Control , the team is now storing everything they need to support and extend an application in one place. This is their “base camp” for release management, allowing them safe and easy rollbacks. Without it, they are always having to log onto VMs to figure out what the actual production config is – and no two environments look like another. Having one authoritative source of truth is a huge leap forward for them. (They’ll go even further when environment configs are also kept in VC, but that’s later.)
With CI/CD – that’s Continuous Integration / Continuous Delivery – they’re eliminating long-lived feature branches and long periods of integration hell and stabilization periods. Smaller batches of work are getting out the door, faster to production. Without this – they’ll be wasting half their time or more trying to make the next release semifunctional, and worse – they won’t be getting the feedback they need to prioritize their work.
With 20% Flow , the team now has at least 20% “free” time to work on projects that interest them. This could be improving their infrastructure or RM tooling, or reading a great book on a new design pattern, or participating in a hackathon. This shows the team is serious about paying down technical debt and investing in their people. This is time the team needs to come up with truly ingenious and creative problem-solving ideas – sharpening the saw, as the saying goes. Without buying out this time every sprint, the team is fully allocated – but has no room to breathe or learn, and burning out at an unsustainable pace, like a jammed highway at rush hour.
With Value Stream Mapping , they take the time to map out exactly how ideas and requirements flow to them and through to production. This is their first step in actually engaging with a stakeholder and putting the problem in terms they can understand – dollars and waste. Everything up to now has really been foundational – mapping out how business value is driven shows the business partner some startling facts on why even “simple” requests take so much time, and exposes to everyone in the room why a better integrated flow of work is needed. Without this, no matter how efficient the team is, they won’t be connected to the actual business problems they’re being asked to solve. They’ll end up treading water. And they won’t have the firepower they need from the business to broker a better partnership with Ops/QA or other siloed teams they’re dependent on.
By itself, Metrics and Monitoring are very helpful. Having developers know and care more about how their systems are actually performing in production – and staying ahead of trouble with a bad release or a failed infrastructure patch – is a huge leap forward. It also can provide tangible evidence Ben can use to check if the process and people investments they are making are paying off in specific, tangible terms. But paired with Hypothesis-Driven Development , and suddenly we’re at another level. We can treat each new feature or “requirement” as being what it really is – a guess – and seeing if our guess was right, or if we need to rethink our assumptions. Without good monitoring, we’re constantly at the mercy of temperamental and whimsical environments. And without valuable business-driven metrics guiding our decision-making, we’re vulnerable to getting hijacked by goldplating or overworked pet projects – or just plain building the wrong thing for the wrong people.
With Family Dinner Code Reviews , the team can better share information and work to improve the group’s collective body of knowledge. Being able to give insights in a productive, safe environment means far fewer bugs being caught early before making it out the door. Done right – respectfully, following playground rules – code reviews can be the single best way of improving quality of code. Having a consistent Definition of Done that the team maintains will also help maintain fair, clear standards – a self-correcting and improving culture of quality. Without it, the same mistakes will be repeated, and more expensive, time-consuming bugs will leak into production, where they’re far more damaging and difficult to fix.
With Blame-Free Postmortems , the team acknowledges that the systems they are working on are unsafe and complex, and that problems never have a human source. By making it safe for people to share the details behind mistakes, the company is better able to learn from them. Without this, the underlying causes of problems will be hidden, and changes will slow as people fear the consequences of making a poor decision.
Shift Left on Testing means the entire team assumes responsibility for quality and any new changes (at a minimum) will have fast-running unit tests proving that it works as designed. By using Defect-Driven Testing – or TDD for newer greenfield projects – Ben’s team chooses to invest in a growing test suite that runs against every check-in. Without a strong unit test base maintained by the coders, integration and functional tests become increasingly more brittle and expensive to maintain – and lag behinds new functionality, meaning changes are either delayed or safety is compromised.
With a Shared Responsibility for Production , Ben’s team begins to rotate direct production support of their applications. There’s simply no substitute for supporting your own code or having a shared work queue and dashboarding so bugs don’t slip behind new, exciting work. This led in turn to the team deciding they needed to better empower their first responder team to assist with triage. First Responder Triage and Runbooks means the team will be spending much less time tracking down false positives and fixable issues, and the developers and first responder teams can work together to improve their reliability and response times for their customers. Without this, technical debt will start building up in hidden corners as the developers race ahead – and the Operations team won’t have the information they need to knock down common problems quickly.
Looking Backward: As the story ends, there’s still clouds on the horizon and a lot of work left undone.
There’s so many other things Ben’s team could have done – how could we have written a book on DevOps and not written more about configuration management, or infrastructure as code? You would think templated environments would have been first on our list. For Pete’s sake, there’s not even much mentioned about containers, Docker, or Kubernetes. What the heck?
There’s no question, those are all good things – in fact for many organizations, they’re essential. But that would come in Year 2, or maybe Year 3, for this particular group. Asking a group that doesn’t have its act together to take on microservices or distributed computing is setting it up to fail. All those great, awesome tools and practices depend on a robust ecosystem and learning-oriented culture that in this case just doesn’t exist yet. It also means putting tooling as the goal, abstracted from the actual business problem that is constricting value. The “unicorns” we’re all so envious of – Microsoft, Google, Netflix – they didn’t start out with these tools or practices. They grew into them, gradually, over time, and each of their implementations looks vastly different.
So yes, their lives aren’t perfect – but the team is definitely happier with their work as it’s gotten more creative and there’s much less friction with other groups and fear of breaking an app in prod. Being loosely coupled as a team and enforcing consistent positive behavior with their partners has helped change the culture of the company as WonderTek has moved gradually toward being learning based.
They’ve also got much better tools/process. Just implementing CI/CD, better config management, and a release process with a weak-but-getting-better automated unit test gating is a huge win for the year. And there’s a much better partnership with Operations with a more effective first responder triage/runbook partnership and pervasive monitoring. They’re using peer reviews to self-enforce quality and share lessons. Security is in its infancy but now a part of the dev life cycle. They’re also turning away more work or abandoning bad bets in the way they gather requirements, use hypothesis-driven development to walk away from failed features / mistaken assumptions early.
Looking Forward: There was a certain amount of navel-gazing as this book neared completion. Where exactly are we going in a few years? What’s the future of the DevOps movement as we stretch into a decade plus of actual hands on practice?
Agile was the hot new thing in the 2000s, which ushered in several development practices and tools that are here to stay – including working in short sprints (the scrum cycle) and more comprehensive use of version control. In the 2010s, DevOps became the new fad du jour, and with it the idea of applying version control and code to infrastructure and more rigorous release management and monitoring tools. It seems likely some new movement will take its place. What will that look like exactly?
It is so shortsighted to say if you follow these practices following this recipe you’ll be successful. I don’t allow people to start telling me ‘we need to do things Agile.’ There’s just no such thing. Talk to me about what you want to achieve, the business value you want to drive, and that’s our starting point. 1
Coming up with a list of DevOps Ten Commandments on a stone tablet might help us sell some books, but it’d be repeating the mistake we made with Agile: thinking that one size fits all. It’s here that our heritage from the efficiency and process-dominated manufacturing world of the 20th century begins to cost us. It’s in our DNA to try to mimic a successful pattern used in a company whose culture we envy, or look for a “best practice” – a proven template to follow step by step. This rigid model simply does not fit in the more fluid and creative world of software development and services delivery. As Bob Lewis so famously said, there are no best practices – only a better one than we had yesterday.2
One of the companies we think is a success story in process is Micron. The task ahead of them is vastly different and in many ways more complex than the ones the Microsoft engineering teams faced; security, the isolated nature of large chip manufacturing plants, and the rigid and horrendously expensive tooling and hardware being instrumented means a more complex operating environment and a need for wise, conservative bets. They are already showing signs of success in the same way that Ben demonstrated at WonderTek:
Nothing innovative or flashy here; this is Theory of Constraints 101 type stuff. First, the release cycle is mapped out from idea to product. This leads to an inefficiency being exposed in the pipeline. A metric or KPI is found that exposes that problem. Once everyone understands and agrees with this assessment, it’s displayed and tracked on monitors, and different improvements are tried until that KPI is no longer the bottleneck. At that point, we move on to the next weak point in the workstream.
In a company like Micron, this has to be done hundreds of times as the ecology is complex; there are dozens of products and services to support. Remembering our limited capacity for change, Micron is treating this as an evolutionary (vs. revolutionary!) process, one workstream at a time.
We have no crystal ball and won’t attempt to play Gypsy foreteller. No one in 2005 could have seen the massive paradigm shift that would be sparked just 3 years later by a few presentations coining this new word “DevOps.” It’s clear from history that the phrase and movement itself began almost by accident. It also seems likely some new word or movement will be used 10 years from now that will take the spotlight. We believe, just as DevOps built on the principles of Lean, Agile, XP Programming, and other movements that came before, that any new movement will follow the same four-step dance we see taking place at Micron: make the work visible, expose the limiting factor, work to reduce or eliminate it, and iterate.
It doesn’t take much foresight to see certain trends continue in their rise. Teams that are guided by the same principles we see in the human body – autonomous, cellular, small-sized, and governed by a central “nervous system” that determines mission and purpose – will become more dominant, as will flatter hierarchies that are more responsive and adaptable to change and less susceptible to the toxic effects of divisional level competition. Microservices and the tools and practices that complement it, including containerization and cloud-based templated environments that are easy to build and replace, will continue to grow in usage. Perhaps most exciting, the tools and monitoring capabilities that allow us to listen to our customers, recover faster from failure, and justify our bets will make quantum leaps in power and flexibility.
In a world that seems increasingly fragmented, selfish, and conflict oriented, the DevOps movement represents a kind of backlash of almost spiritual idealism; how can we get people to work together for the common good? The software may change, but the qualities we’ve discussed in this book will remain timeless: humility, progressive learning, inclusion, honesty, service-based leadership. Our respect for others shows in the care we take in exposing waste and toil, and the amount of effort we are willing to invest into eliminating waste through automation and better collaboration.
So I perceived there is nothing better than for people to enjoy their work, because that is their reward; for who can show them what the future holds? 3
That inborn, spiritual need to enjoy our work is what sparked this movement – whatever we choose to call it down the road. We believe this very human-based inherent need will become more vital and important in the decades ahead. Enterprises that empower their teams and allow people the freedom to innovate, create, and improve in the way they do work will continue to flourish.
We’re starting to realize that software development isn’t the same as manufacturing. There are definitely some principles that are shared with manufacturing but it’s a different beast. In manufacturing we build identical things. In software, we never build the same thing twice. When we try to approach software development through a manufacturing lens, we limit ourselves. The effective ones of us will realize that there is no “best practice”, only “better practices” – and they change. What works effectively depends on the software, its criticality, its technical debt, its lifetime, your team, their leader, and their interrelationships. The only people capable of identifying a practice that works well for a team, is the team itself.
At the end of the day it just depends. Because it always depends. Each team is different. People are different. Leaders are different. Technology is different. Companies are different.
That being said, I think these principles are the ones that will guide the winners over the next decade.
Success comes from:
Small teams that build deep relationships and determine how they work.
We strive to serve our customers and not just sell or deliver to them.
We don’t take ourselves too seriously, because we all have much to learn.
We take care of ourselves like we take care of our customers.
We balance life with work and we strive to live authentically and wholly.
Whenever possible we have more people contribute to finding a solution than less.
We don’t cheat safety, quality or security for shorter term gains.
Whenever possible, we choose to learn and experiment and make mistakes, because in the making of mistakes we learn the most.
We learn that people are more important than work. Our families are more important than work.
Life is about relationships: loving God, loving each other, loving our families.
You only have the one life to live, make sure to spend it on what’s important.
Now… Go and be marvelous.