Like many people, we are excited about the adoption of Agile (adaptive) project management and development in general, and about the adoption of Scrum in particular.
Unlike many people who are so dogmatic about Agile or Scrum, however, we do not believe that the current corporate world has been reorganized completely for Scrum. Maybe some commercial software companies have done that, but the large majority of corporations have not fired all their specialists or given them a new generalist title as some Scrum dogmatists believe they should. Most of these companies still have their current organizations the way they are, with many separate functions in their IT departments. CapitalOne, one of the companies that has adopted Agile often cited by many Scrum trainers, is still hiring business analysts, business systems analysts, and project managers. This is also true of many large companies known for having transitioned into Scrum, such as Sabre, Verizon, NBC Universal, General Dynamics, Texas Instruments, and American Airlines.com, just to mention a few.
We will talk more about Scrum’s origin and fundamentals later in this chapter but for now, let’s say that Scrum, as in the sport of rugby, is a way of restarting the rugby game, either after an incident or after the ball is out of play. The idea is to keep the game (software development) rolling.
Even though Scrum can be used outside of software development, we will focus in this book on Scrum as an Agile process framework for software project management and development alone.
Having managed projects in the traditional command and control style for a long time, we have both witnessed, time and again, that Agile and Scrum seem to help our teams produce software results more effectively than their command and control counterparts.
This is not to say that the traditional plan-driven style no longer works in any circumstances; there is much rigor in the traditional approach that even Agile and Scrum can benefit from. What we are simply saying is that we can do better if we know how to take a few steps back to reflect on what we do and learn to improve our techniques and processes with new ideas and concepts.
Because people and companies tend to embrace change slowly, the fact that the Agile movement, and especially Scrum, has taken off and generated this much excitement is a very good sign. Together, we hope to contribute to spreading the adoption of Agile and Scrum with this book, which is derived from many years of experience in software project management and development in real-life.
Like many of us, if you can get past obstacles, you will find Agile and Scrum amazingly refreshing and you will learn how to successfully implement them on a project. Otherwise, your road can be littered with difficulties, which ultimately may cause your project to fail.
There are many reasons why teams sometimes fail with Scrum. One of the most common reasons is that many professionals are still unaccustomed to Scrum, even after taking a Scrum class or reading a few white papers about it. The second reason may be that their project is so complex that more advanced techniques are needed to reduce the project’s complexity and get it under control. A third could be that their organization is not yet set up for Scrum, or that the teams do not know how to use Scrum within a company’s current constraints. This could either be because of lack of experience or because the team has been ill-advised by dogmatic ScrumMasters or coaches, something we address in this practical guide.
Whatever the reason, Agile and Scrum come with change, and change is always difficult unless properly managed. But with good management these difficulties can be turned into opportunities.
One of our goals in this book is to prepare you for the changes you and your team will go through with Agile and Scrum. We want to help you make the process as smooth as possible while giving your team a chance to adapt to, embrace, and succeed in your Scrum project.
The foundations of Agile Software Development and Project Management are, without a doubt, the Agile Manifesto and the Declaration of Inter-Dependence.
In 2001, a group of software experts got together in the Snowbird resort of Utah to draft what is known as the Agile Manifesto (www.agilemanifesto.org):
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.”
[© 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas.]
Along with these four values, the Agile Manifesto has twelve principles:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity—the art of maximizing the amount of work not done—is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Although the Agile Manifesto was drafted in 2001, a few years after Scrum was announced at Object Oriented Programming, Systems, Languages, and Applications (OOPSLA) in 1996, it is a well-known fact among experts that it has great influence on Scrum. This influence was obvious in Ken Schwaber’s second book, Agile Project Management with Scrum, in which he wrote that Scrum is one of the Agile processes with values and principles as described in the Agile Manifesto.
While the Agile Manifesto dealt with software development, the Agile Project Management “Declaration of Interdependence” which another group of experts pulled together in 2005 focused more on the project management side (http://pmdoi.org):
“We are a community of project leaders that are highly successful at delivering results. To achieve these results:
We increase return on investment by making continuous flow of value our focus.
We deliver reliable results by engaging customers in frequent interactions and shared ownership.
We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
We boost performance through group accountability for results and shared responsibility for team effectiveness.
We improve effectiveness and reliability through situationally specific strategies, processes and practices.”
[©2005 David Anderson, Sanjiv Augustine, Christopher Avery, Alistair Cockburn, Mike Cohn, Doug DeCarlo, Donna Fitzgerald, Jim Highsmith, Ole Jepsen, Lowell Lindstrom, Todd Little, Kent McDonald, Pollyanna Pixton, Preston Smith and Robert Wysocki.]
Whether the Agile Manifesto and the Declaration of Interdependence (DOI) came to the experts’ minds first or after they had been somewhat influenced by Scrum or by any other then existing Agile processes, it does not really matter.
What matters is that if your truly understand the meaning of the Manifesto and the DOI, you will have a leg up in adapting Scrum, should the need arise, without betraying its Agile foundation.
Historically, the term Scrum comes from an article published by Hirotaka Takeuchi and Ikujiro Nonaka in the Harvard Business Review in 1986. In that paper, entitled “The New New Product Development Game,” Takeuchi and Nonaka described a holistic approach in which project teams are made up of small cross-functional teams, working successfully toward a common goal, which the authors compared to the Scrum formation in rugby.
While working on building an Object Oriented Analysis and Design (OOAD) tool at Easel, Jeff Sutherland, then VP of Engineering at Easel, Inc. realized that his software team would need an enhanced version of rapid application development. What he wanted was a process similar to Scrum where at the end of short iterations, the CEO at Easel would see working code demonstrated, rather than paper Gantt charts.
During more or less the same period, Ken Schwaber (see recommended further reading in the bibliography) was actively looking into how he could help his company, Advanced Development Methods, Inc. (ADM), enhance their software process in order to improve their teams’ productivity.
After further analyzing how other successful independent software vendors (ISV) built software, Ken came to realize that all their development processes were similar in that they all used empirical processes, which requires constant inspection and adaptation.
At the request of the Object Management Group (OMG) in 1995, Jeff and Ken worked together to summarize what they had learned throughout the years; they created a new methodology, which they named Scrum, and described in Schwaber’s article, “Scrum and the Perfect Storm,” at www.controlchaos.com/my-articles.
Notice in Figure 1.1 that the Scrum team, which should be a cross-functional team, is composed of a ScrumMaster, a product owner, and the development team (or simply, “the team”), with all the skills needed (such as requirements gathering, designing, coding, and testing) to build the software product.
Even in the best situations, when the Scrum team composition is fully cross-functional, it does not necessarily follow that the Scrum project members are, organizationally and hierarchically, all part of the permanent Scrum project team structure. Unless your company is a resolute proponent of enterprise-wide Scrum deployment and has completely reorganized itself, following Scrum organizational structure recommendations, Scrum teams are still borrowed from different organizations to which they belong, such as QA, Enterprise Architecture, Pre-production, or DBA.
Even some of those companies often cited as having adopted Scrum still have Project Management, QA, Pre-Production, Business System Analysis Groups, and Enterprise Architecture working as separate functions in IT, and project teams have to negotiate with these groups to get their Scrum projects going. Very rarely does a company fire all its project managers (Scrum does not have a project manager per se) or change their title to ScrumMaster or reorganize all its separate IT groups around multi-disciplinary Scrum project teams. That would be the dream, but it is not the current reality yet.
In Figure 1.1, you can see how a Scrum project gets started. As you can see in the diagram, it all begins with the product owner who is responsible for taking input from different stakeholders, or users who represent them, to elaborate a list of requirements to create a product backlog.
Simply put, the product backlog is a prioritized list of requirements, which can include everything from business features to technologies to technical issues to bug fixes.
Some practitioners and authors, such as Henrik Knitberg in his book, Scrum and XP from the Trenches would prefer to keep the product backlog at the business level to include only business requirements, as we do.
As you will learn in Chapter 4, “A Visual Requirements Gathering for the Product Backlog,” the user requirements for Scrum product backlog are usually gathered as short user stories during a one- or two-day requirements workshop prior to the release and Sprint planning meeting.
While release planning in Scrum was somewhat optional in the early days, it has proven to help many Scrum teams become even more effective throughout the years. Thus, we strongly recommend that the product owner go through release planning with the team even if this is a difficult exercise, as it will require you to learn about the product prior to the planning meeting.
The more you, as the product owner, know the product, the more you can help the team. The key goal of release planning is for the Scrum team to identify all the releases the software product should have, along with a probable delivery schedule. Normally, release planning should last four hours for a Scrum team with four-week Sprints.
Besides release planning, the Scrum team should also go through some Sprint planning, either as part of the release planning process or independently after the release planning is done.
Normally, the Sprint planning meeting should be around eight hours for Sprints of four weeks and should be adjusted to four hours for Sprints of two weeks.
As a common practice, the Sprint planning meeting should be divided into two equal four-hour meetings.
During the first part of the meeting, the product owner will go through the requirements, as user stories, to decide, with the team’s feedback, which ones should be part of which Sprints and what their goals are. The first of the two-part meeting is mainly to answer the “What” question.
During the second part of the Sprint planning meeting, which focuses on the “How,” the Development team will try to identify tasks from the previously chosen stories and deduce how much time (in hours) it will take them to turn these tasks into potentially shippable product increments. Unless the team uses some kind of planning software, all the development tasks that are part of the Sprint will normally be consigned to a Task Board, some kind of white board on a wall, for easy team allocation and tracking.
As soon as the release and Sprint planning meeting is done, then starts the actual sprinting work (Figure 1.2) along with its 15-minute daily Scrum, or Daily Standup.
Originally, the Daily Standup could last up to 30 minutes, but as part of Scrum evolution, or adjustment, which we will talk more about in Chapter 12, “How to Adapt Scrum (Without Destroying Its Agile Foundations or Doing Negative ScrumButs),” its duration has been reduced more and more in practice, to around 15 minutes today.
Normally, the duration of a Sprint will be from one to four weeks. Except under very special circumstances, no additional items should be added to or deleted from the Sprint backlog while the Sprint is underway, unless the team and the product owner agree to them, but this is something like an exception rather than the norm.
Unlike the traditional process where the project manager is responsible for organizing weekly status meetings to track project status, with Scrum, the team will get together every day to inspect, not the project status, but the team’s progress toward the Sprint goal. This is the reason you often hear people say that the daily Scrum is not a status meeting.
To keep track of the team progress towards the Sprint goal, a burndown chart will be created by the team to show how much work remains until the team is done with the Sprint. Even though the creation of this burndown chart is the team’s responsibility, it can be updated by the ScrumMaster whenever the team does not have time to do so.
Just before the end of every Sprint, the team will meet with the product owner, as part of Scrum’s Inspect and Adapt mechanism, to go through what is known as a Sprint review organized by the ScrumMaster (Figure 1.3). This is another time-boxed meeting, which normally lasts four hours for a four-week Sprint or two hours for a two-week Sprint.
The objective of this meeting is multi-fold: The first is for the Scrum team and the product owner to discuss what was done and what was not done. The second is for the team to demonstrate what was built to the product owner and get her feedback. Finally, the third objective is to get updates from the product owner regarding any new changes to the product or market direction.
Right after the Sprint review and prior to the next Sprint, the Scrum team will also meet to go through a Sprint retrospective to identify what worked and what did not work during the current Sprint. The intent is to see how they could make their collaboration even more effective going into the next Sprint.
As a common practice, the retrospective meeting should normally last three hours for a monthly Sprint but its duration should be adjusted, as is the case for all the other timed-box meetings, in proportion to the length of the Sprint, such as two hours for a two-week Sprint.
Figure 1.4 provides an overall graphic of the collaborative responsibilities of Scrum team members.
Overall, Scrum does not sound too complicated as a general project framework, right? Yes, but while Scrum sounds simple in theory, it can be very difficult to implement, especially if your company is still new to Scrum practice or if you want to turbo-charge your Scrum project with some of the current practices.
Even though Agile and especially Scrum can be difficult to implement, it has proven to be extremely effective when properly deployed.
As you read this book, you will see many reasons why Agile and Scrum are normally more effective in project management and development. There are, however, four advantages that we would like to emphasize up front:
A systematic risk reduction mechanism: All of us who are responsible for project planning and execution know how important it is to reduce the level of risk or uncertainty to zero or to the lowest level possible.
While there are as many as four different ways to deal with risks (avoidance, transfer, accept, and mitigate), project managers often end up having to mitigate risks in the end. This is where Scrum excels with its frequent Inspect and Adapt cycle.
A leaner software development life cycle:
In Figures 1.5 and 1.6, we see the large difference in timeline, with one of the teams using a longer life cycle (Figure 1.5) while another team is using a leaner life cycle (Figure 1.6), making more efficient use of productive time.
A more adaptive project management process. Unlike the sequential process used within the waterfall environment, which considers project stability as the foundation (Figure 1.7), Scrum will look more like Figure 1.8, in which change is considered to be the only constant.
A project management and development process framework based on people’s motivation and pride. More than anything else, this may be one of the most powerful tenets of Agile and Scrum. The new focus is not on having the manager dictate to team members what they should be doing, but to let the teams decide for themselves how they would go about accomplishing their work on their own.
Scrum proposes a new software management framework, which is based on a project team’s self-organization, motivation, ownership, and pride in their achievement.
This introduction to Agile and especially to Scrum, although short, will provide you with enough understanding to be able to move forward with the rest of this book.
But before doing this, let’s also say that since our focus is on Scrum, we will, therefore, refer only to Scrum for the rest of this book.
The Agile was born in 2001 when a group of experts got together in Utah to draft what is now known as the Agile Manifesto, the focus of which is on the software development side.
As a complement to the Agile Manifesto, another group of experts came together in 2005 to draft another document called the PM Declaration of Interdependence (DOI), the emphasis of which is on the project management side.
Together, the Agile Manifesto and the DOI form the foundation of the Agile movement and of all Agile development and project management processes, including Scrum.
Since Scrum cannot be used in many contexts without some sort of adaptation, a good knowledge of the foundation of the Agile movement and processes will enable you to know how to adapt Scrum to your environment.
While you will see all the advantages of Scrum during the course of this book, four of them are worth pointing out right away: (1) a risk reduction mechanism, (2) a leaner software process, (3) a more adaptive software project management process, and (4) a framework based on a team’s self-organization, motivation, ownership, and pride.
Since the focus of this book is on Scrum, we will refer only to Scrum from now on throughout this book.