5.

IMPLEMENTING AGILE AT SCALE: MICROSOFT

The point is that the conversation takes place and it’s a safe conversation to have.

—AARON BJORK

Not long ago, Brian Harry, corporate vice president of Microsoft, received an unexpected complaint on his blog. Is Microsoft too focused on Agile? “It is obvious,” a reader named Kasper wrote, “that you have focused on agile tools the last six months. . . . My question is how long are you going to focus on this area?”

It may come as news to some that Microsoft is focused on Agile at all. The image that many people have of this giant corporation with 2016 revenues of $85 billion and 114,000 employees is that of a giant battleship that is strong and powerful but slow to maneuver and not always customer-friendly.

In fact, that was the image that came to mind when Aaron Bjork, a group program manager in Microsoft’s Developer Division, contacted me and offered to share the story of Microsoft’s Agile transformation journey with the SD Learning Consortium.

The Developer Division consists of about 4,000 people, working in hundreds of teams, each team having ten to twelve people and working in three-week sprints. Bjork’s specific group is Visual Studio Team Services. This group comprises forty teams totaling about 500 people. Bjork himself looks after seven of those teams.

The Developer Division ships a range of products and services, including Visual Studio, Visual Studio Team Services, Team Foundation Server, and TypeScript. Other groups at Microsoft, like Windows, Office, and Bing, are separate, but all of them are in various stages of an Agile transformation. The Developer Division is leading the enterprise-wide journey to become more Agile. It owns the “first party engineering system charter” (1ES) and is driving that across the company. There are monthly scorecards on how the big divisions are doing in adopting it.

We ended up having several site visits, and we found that Microsoft was less like a giant battleship and more like a flotilla of speedboats operating and maneuvering in an orchestrated fashion.

In July 2011, when Microsoft corporate vice president Brian Harry announced a corporate commitment to Agile in a blog post that was almost a love letter to Agile, many people were skeptical. Was it conceivable that a giant firm like Microsoft could embrace Agile management?¹

Ken Schwaber, the cofounder of the Agile methodology known as Scrum, spoke for many when in a blog post he questioned whether a big corporation like Microsoft would ever be able to emancipate itself from viewing people as “assignable, parsed, optimized resources.” Just as only a very small percentage of U.S. adoptions of lean manufacturing reflect its core principle of “respecting, valuing, and engaging the workers,” would the commitment to Agile announced by Microsoft be a commitment to the outward forms of Agile, but without its values? Would this be the same old “predictive manufacturing model, wrapped in Scrum tools,” that would be unable to generate creative, sophisticated, quality products?²

Happily, we can report from our site visits that Agile values are alive and well in the Developer Division of Microsoft. The division is not only implementing Agile practices for itself, it is also promoting them for others at Microsoft. Moreover, everyone we talked to—including developers with whom we had unscripted conversations—is living, thinking, talking, and acting with Agile values. We found a pervasive Agile mindset in which respecting, valuing, and engaging those doing the work in response to customers’ needs is central.

Our host for the visit, Aaron Bjork, plays golf as a hobby—an unusual choice for a software developer. He also looks after a small farm with his family in his spare time. He was a business major in college, studying finance and information systems at the University of Washington. He gave up the finance path because, as he says, he had too much energy and creativity. Finance would have bored him to death. He joined Microsoft in 2002 and started working as a coder. Seven years ago, he became what Microsoft calls a program manager, which is essentially a “product owner” in the terminology of Scrum. Why did he change? He used the analogy of a car to describe his passion. “I enjoyed having the hood up and getting my hands on the engine—getting dirty,” he says. “But my real passion is deciding what kind of car we’re going to build, and having input into the ‘little things,’ like the color of the leather stitching.” Bjork knows these things matter to customers. That’s his passion.

“My job,” says Bjork, “is listening to what customers want and making sure that we build them something they need, and something we can sell, differentiating between wants and needs. And then finally apologizing for all the things we are not doing. There is always too much that we could be doing but can’t get to. We just can’t do it all.”

The transformation at Microsoft took time. Aaron had begun experimenting with Agile and Scrum with his previous team back in 2008. About a year later, several teams began implementing Scrum, and there were other pockets of Agile in various places around Microsoft. In 2010, the Team Foundation Server team decided to “go Agile,” with all their teams operating with Scrum practices and roles in three-week sprints, in the same cadence. Based on the success of these efforts, in July 2011, Brian Harry, the corporate vice president, publicly announced in his blog the commitment to Agile and Scrum in the Visual Studio group. In late 2011, the entire Developer Division decided to “go Agile.”

By July 2015, when we first visited Microsoft, Visual Studio Team Services was in its first week of sprint #87. In May 2016, we encountered the second week of sprint #101. The three-week sprints continue in a steady rhythm, through holidays and corporate changes, come what may. Over the holidays, it’s just a lighter sprint. The teams like the rhythm. They like the order that it brings.

But the Agile transformation journey has been anything but a straight path from A to B. Introducing the practices of Scrum—sprint planning, backlogs, daily standups, retrospectives—was only part of the challenge. More important—and difficult—was the shift in mindset for all involved.

It’s a journey that never ends. “The journey,” says Aaron, “is neither a train wreck nor a tale of unbroken triumph. It’s had its ups and downs. We did some things right and some things wrong. There are places where we have improved.

“Initially there was a lot of pain. It took a long time before we could actually ship at the end of a three-week sprint,” he says. “In reality, we were running three-week milestones. That’s all they were. We would get to the end of a sprint and a team would claim that a feature was done and be celebrating and then I would try to use it and it wouldn’t work. The team would say, ‘Oh, we didn’t do the setup and upgrade for it.’ And I would ask, ‘I thought you said it was done?” And they would reply, ‘Well, yes, it’s done. We just didn’t do the setup and upgrade.’ It took a long time for everyone to grasp that we needed to get fully done in every sprint. It took about a year to learn how to do it,” he says.

“For instance, let’s not be date-driven. Let’s be value-driven. Let’s ship the product when it’s ready. We have learned not to change ship dates based on external events. If there is a problem, the team fixes it if they can. If they can’t, they just don’t deploy that feature. It’s the team that decides. The team does its own stress-testing and documentation. It felt awkward and even weird for a long time,” says Bjork. “For a lot of people, it was very disruptive. Now it’s part of our DNA. It’s just the way we work.”

Here are the keys that Bjork believes are needed to make Agile happen at scale.

Get the Right Balance of Alignment and Autonomy

The goal at Microsoft is to have alignment at the top and autonomy at the bottom. “The teams need autonomy,” says Bjork. “That’s what puts a spring in their step coming to work to deliver great value to customers. But, at the same time, their work has to be aligned with the business. If there is too much control, nothing gets done: No one wants to work there and it’s not a fun environment. In fact, it’s a disaster. If there is too little control, it’s chaos. Everyone is building whatever they want. There are no end-to-end scenarios. Customers are frustrated. Nothing makes business sense. The managers are always striving for the right balance.”

The leadership team is responsible for the rules of the road. This includes clarifying the roles, the teams, the cadence, the vocabulary, and the limits on the number of quality problems—bugs—a team can have (“the bug cap”).

The team has autonomy in terms of how they go about doing the work in both planning and implementation. Within the overall framework, each team can take a different approach. The specific engineering practices are up to the team. For instance, whether the team decides to do pair programming is up to them. (By comparison, at Menlo Innovations, discussed in Chapter 2, pair programming is compulsory.)

“The role of management is akin to establishing the rules of the road,” says Bjork. “You can drive down the freeway fast. In fact, the freeway is designed to help you go fast. But there are rules. You must use your blinker when you cross the dotted line. You must slow down at certain points. The authorities could make the freeway much safer by putting speed bumps every hundred yards and a stop light every mile. It would be safer, but it would slow things down. At Microsoft, we take the same approach,” he says. “We are specifying the minimum basic road-rules the teams need to adhere to. Our goal is to ensure that the rules help the teams move fast, to get where they want and need to go, not just slow them down.”

Of course, when Bjork asks an engineering team, “What does leadership make you do that is slowing you down?” he gets a ton of stuff offloaded onto him, not just one or two things. “They often pull out a whole scroll of issues. They were just waiting for me to ask! They tell me everything I’m doing wrong and we have a conversation. The point is that the conversation takes place and it’s a safe conversation to have.”

Master the Role of the Agile Manager

What happens when a team fails to deliver what is expected in a sprint? A manager doesn’t monitor a team’s burndown chart of things-to-do. “The burndown chart is for the teams,” says Bjork. “If they get behind, guess what? We talk about what to do. That’s the behavior we want. We get that behavior because the culture supports it. If I yell at the team or monitor their burndown chart, guess what I get? Perfect burndown charts. I have to decide: Do I want perfect burndown charts or the right conversation? In the end, it has to be the latter.”

It’s the difference between mindset and practices. It’s vital that people understand why they are doing the practices and then taking responsibility for the value that is meant to come with them. If the daily standups aren’t working, then be a grown-up. Make a change! That’s where autonomy comes in. Say Bjork, “You are in control! You are responsible!”

Handle Dependencies at the Team Level

In Microsoft’s Developer Division, dependencies are handled to the extent possible by the teams themselves. The teams all know what the other teams are doing. They are all in it together. It’s not like the manager or the team works only on their piece of the product and that’s all they care about. They all know what other teams are doing. If one team has a dependency on another team, they are not going to wait till a meeting to let them know about it. The program managers and the engineering managers are talking about it and figuring that out ahead of time. They are self-managing and learning how to become good at it.

Of course, it sometimes happens that the team leaders get to the meeting and realize, “Oh, you guys are starting this item already? We didn’t know that! You should talk to the other team.” They make sure that there’s a conversation and it’s taken offline and they figure it out.

It’s expected that the three-sprint plans include all dependencies. Bjork works with his seven teams. And the same process is happening in the six other groups of teams. The managers are talking and sorting it out on a continuous basis. They sit in the same team room. It’s an ongoing conversation.

Every three months, there’s a standing meeting across all of the teams. It’s called “a feature team chat.” Every team comes in and shares their plan. Bjork has ninety minutes. His teams come in and they each have ten to fifteen minutes to share their plan. And his peers come in with their teams and they also share. This takes place across all the teams so that everyone is aware. This regular “ceremony” provides the leadership team of the Developer Division a chance to stay in sync with what’s going on.

Ensure Continuous Integration

Continuous delivery has required more modularity in design and a change in the architecture of the software. When the Developer Division first got into software as a service, they put it in the shared-computing space known as “the Cloud” and crossed their fingers. It didn’t work. When one part of the product failed, the whole product failed. They saw that they needed each part of the service to be able to fail independently of every other part and not cascade through the whole product. There were deep architectural changes needed to support that.

When they started, each team would work in a branch of the code during a three-week sprint. At the end of the sprint, they would put it all together, and it was chaos. In fact, the teams had created a lot of “integration debt.” That model didn’t work. To ship every sprint, they had to make a fundamental change.

In principle, all the teams are “working in the same branch.” What that means is that each of the teams is branching their changes in a program called Git. But the teams don’t work in a silo by themselves for three weeks and then hope it all comes together. They come together all day, every day. If a team breaks the build, it fixes the build immediately. They do whatever they have to do. The longer the team waits to put code together, the greater the risk of technical and integration debt—and disaster.

The teams use what they call “feature flags.” Here’s a high-level description of how it works. If they are going to do something new, the very first thing that they do is to isolate the code that they are changing and build a switch into the code. It is powered by a flag in the database. It’s a configuration change. When the team writes code, they write it behind the safety of the flag. At some point, when they feel it’s ready, they can turn it on just for the team. That switch is not a global switch. It’s a switch for an account in the system just for the team. If that goes well, then the team can turn it on for certain customers. Those customers can see it and try it. They help the team find bugs and problems. When that’s done and the team is really ready, the release notes are prepared and the team announces that they are going to flip the switch for everybody. Then they go back and refactor the old code out. This enables the teams to work alongside each other on the same code without breaking one another’s work.

At the end of every sprint, the team sends out an email to all 500 people in the Visual Studio Team Services group and the leadership team. They talk about what they accomplished during that sprint and what their plan is for the next sprint. And they record a three- to five-minute video. (Warning: The videos can get fancy if the teams have aspiring Hollywood directors.) The video replaces the print demo.

Keep on Top of Technical Debt

“In the old days,” says Bjork, “when the code had been written, the team had a party. They were celebrating. They felt they had accomplished something. But in reality, they were sitting on a mountain of bugs. In fact, they hadn’t even found all the bugs. The team then had to go back and find them all. And fix them. And they were still months away from delivering software. It was a nightmare.

“Now the number of bugs never grows,” he says. “The bug cap is the number of engineers on the team times four. So, if you have ten engineers, your bug cap is forty. If you get to forty bugs, the team stops work on new features and the next sprint and gets the bug count back down below forty. It’s self-managing. Teams know this. It means that we can ship product all the time because we know we’re always in a healthy state.”

Embrace DevOps and Continuous Delivery

In this way of working, known as DevOps, development and operations merge. The teams own the planning, execution, delivery, and operation of each new feature.

“If the service goes down,” says Bjork, “then the team has to stop everything and deal with it. If bugs are found or fixes are needed, the team is responsible for fixing it. They used to have a separate support team doing that, but who wants to spend their time fixing someone else’s blunders? End-to-end responsibility for quality helps keep the team committed to quality. The teams own the entire life of the feature. If the service is breaking down frequently, then it’s a problem with the quality of the code. It’s also a motivation to write great code. The teams are living quality on a continuing basis. They own the features they create. There’s no one else to blame. And they don’t have the pressure of one big release. They can stage things and resolve problems as they go.

“The change in time frame makes a huge difference. A deadline now is three weeks,” says Bjork. “Three weeks is no big deal. Before, you had only two opportunities and if you missed them, you had to wait two years. Now if it’s not high quality, you don’t push it out. You hold it. It’s disappointing that you didn’t get it out. You talk about it in your retrospective. Did you do something wrong? Or did you just underestimate the level of complexity? Did you miss something? It’s better to have that conversation than to have a fire drill and punish the team for not delivering what they promised, or worse, pushing out a poor-quality product.

“It’s a huge change in how things happen,” he adds. “Before, there was a lot of punting bugs around the organization. People pointed fingers as to who was responsible for which bug. Bugs lasted a lot longer and there were more of them. Now the team itself finds them, fixes them, and is done with it. The lead time on bugs has come way down.”

Two years ago, the group flirted with the idea of daily delivery of code. But they found that customers didn’t even want it. It meant too much change. There was no business need to do daily delivery. At the same time, the teams are learning to work in ever smaller chunks of work.

Continuously Monitor Progress

The teams regularly monitor how the features are being used. The results flow into the aspirational backlogs, which are called scenarios. Every month, the program manager reports out on metrics of the accounts, measuring different aspects of the service. The group is learning to become a “data-informed” business. They don’t call it “data driven” because that would run the risk of missing the big picture. They use their brain and their gut feel as well as being informed by the data. The data isn’t an afterthought, though. It’s often the first part of the conversation.

Part of the very definition of “done” is having the right measures. The teams see this data and monitor it, both when they are testing it and as soon as it goes live. It’s not something they do after they ship it. It’s part of the acceptance criteria to ship.

As soon as the code ships, the team asks: How are people using it? Is it driving people through our conversion funnel? Are they becoming a dedicated account or just casual users? They use the metrics to drive the business forward.

Listen to Customer Wants, but Meet Their Needs

The program managers do all kinds of customer visits, including at the highest level at the Microsoft’s Executive Briefing Center—which is all about strategy—in customer councils with folks who use the products, and regularly with the Microsoft consultants. The program managers are talking to customers on Twitter all the time.

The teams don’t blindly follow what customers say. They follow what Bjork calls “the cookie principle.” “If you have a plate of cookies and you ask people if they want one, they will say yes. No one turns down a cookie. If you go to a customer and ask them, ‘Do you want this feature?’—guess what they say? Of course, they say yes. Why wouldn’t they? It’s the innovator’s dilemma,” he says. “There’s a whole bunch of good stuff out there that you can do. You need to listen, but not blindly follow. The program managers need to listen to what the customers say they want, but their job is to build them something they need. And something that the firm can sell. Otherwise the managers are not doing their job.”

Deal with Directions from Above

Bjork has never heard anyone above him tell him to stop doing Agile management. One reason is that the Agile teams have been very successful. Agile has grown and spread through its success.

“There is very little load balancing among teams,” says Bjork. “If a team gets behind, we don’t break up the team or move individuals to the team to fix it. We ask the team itself to fix the problem. We try to keep the teams together for twelve to eighteen months. That’s what the teams themselves like. The goal is to let them get good at building software together. That’s their job. If we’re reshuffling them every three sprints, or even reshuffling the things they are working on, that makes it pretty hard to get really good. The firm is making an investment in the team for at least nine months or a year.”

Use Self-Forming Teams to Encourage Team Ownership

The managers let people choose periodically which team to work on.³ Around two-thirds of the team members decide to stay where they are. As a result, there are not many brand-new teams. But the team members have a choice. The result is a significant investment in persistent teams, which, quite apart from team well-being, leads to higher performance.

The team owns the backlog of things-to-be-done. “Of course,” says Bjork, “there’s a lot of discussion about priorities. But a manager doesn’t tell the team what to do next. Nor does the team dictate to the manager. They talk about it all the time. It’s a constant conversation. It’s like, ‘Hey, what about this? Should we be doing that? Do you think we’ve already invested a lot there? Should we go over to here?’ And I as the program manager am having the same kind of conversation with my leadership team.

“These conversations require significant mutual trust,” says Bjork. “As a manager, you can’t be involved in everything and you can’t know everything. A manager says things to the team and the team listens, but they don’t blindly follow what the manager says. It’s give-and-take. It’s a data-informed conversation. That conversation goes on all the time.”

Recognize the Team Is the Product

In software, the product life cycle is shortening. “In traditional accounting,” says Bjork, “the business asset is the product. But more and more, in practice, the asset is a team that is capable of delivering products. The team has a longer lifetime of generating value than the product itself. That’s a big shift in focus.

“Microsoft has an advantage in that it embraced teams long before it embraced Agile,” he says. “There was already a strong team culture. It’s more difficult for firms becoming Agile if they don’t have a history of teams.”

Microsoft sees itself as making an investment in its people. Sometimes organizations don’t recognize that investment. There’s a risk that the top may think of people as just pluggable resources. “At Microsoft, it doesn’t work that way,” says Bjork. “The leadership team understands that.”

Build Quality from the Beginning

There was a lot of learning at the start of the Agile transformation at Microsoft. “In the first sprints,” says Bjork, “there was agreement on doing three-week sprints. The leadership signed off on the idea of Agile, but they were anxious as to how it was going to work. They planned for ‘a stabilization sprint’ after five sprints. However, that encouraged some teams to think, ‘No need to worry about bugs, because we have the stabilization sprint!’ A lot of bugs were generated and all the teams had to pitch in to help fix them.

“In effect,” he says, “we had told people to do one thing, but we created an environment that prompted some teams to do the opposite. Who could blame them? The teams told us, ‘Don’t ever do that to us again!’ It was an example of unintended consequences.”

Above all, the goal is to avoid the sequence: Write code in the first sprint. Test it in the second sprint. Fix bugs in the third sprint. The rules of the road are: Deliver finished product every sprint.

It’s part of the concept of autonomy. If the teams can control their own quality, they’re not surprised about what they’re going to have to do in the future. Like working on weekends. They can take care of their own business and not be subject to things they can’t control.

Use Coaching Carefully

External coaches and trainers at Microsoft were noticeable in the site visits by their absence. There had been some coaching and basic training in Scrum at the outset. But after a while the group just started “doing it” themselves, figuring out what was working and doing more of that, and seeing what wasn’t working and not doing that. Some of the Microsoft staff and managers in effect became Agile and Scrum coaches. But overall, the group itself just set out and went for it.

More recently, there’s a recognition that a lot of new people have come on board who didn’t do the basic training. So, thought is being given to doing more training. At the same time, there is a recognition that there is no “one size fits all” and that what works elsewhere may not fit the Microsoft culture.

Ensure Top-Level Support

Microsoft’s top management was cautiously supportive about the Agile transformation at the outset. But that has changed. “There is now a broad recognition,” says Bjork, “that Agile is the modern way to build software. That’s not too difficult at the team level. You grab ten people and do some training and you can do it. But how do you do it across 4,000 people and stay in sync? That’s the challenge. How do you do it on a large scale?”

To achieve that, the support of Corporate Vice President Brian Harry has been central. Bjork has had the benefit of working in the Developer Division where Scrum and Agile practices now have a deep foothold. “It also helped,” says Bjork, “that our customers were writing user stories for us almost before we even knew what user stories were. We had to be fast learners.

“It takes time,” says Bjork. “We are now seven years into this. We didn’t make these changes all at once. The physical space change is one of the last changes we’ve made. If we had moved into a team room, put all the disciplines together, and tried to do three-week sprints, it wouldn’t have worked. Agile has to evolve. Emotionally, it takes time. You can’t make that change all at once.”

An Agile transformation of an existing business is one thing. But what about creating a completely new business that creates whole new markets? It is to this issue that we now turn in Chapter 6.