Our experience is that there has been a great deal of improvement in projects during the past decade. To be more specific, we’ve seen the Agile movement make an impact on how projects deliver business value.
The focus of this book, when it comes to processes and frameworks, is on Agile methods such as Scrum and XP. The reason is simply that Agile fits nicely with the concept of ALM.
This chapter looks at how you can use Scrum as an Agile project management model to deliver software. We cover the Scrum process in depth, including how you can use Scrum and Agile practices, such as Agile estimation and planning, in combination with an ALM tool. This chapter gives you insight into why Agile and ALM are a good match.
This chapter also shows you some other Agile processes that are popular. Scrum is a great framework, but for some projects or some organizations, you may need another process to help you run your projects.
The Scrum Framework
Next we examine one of our favorite development models: Scrum. With all the attention Scrum has been getting in recent years, you may be misled into believing it’s a fairly new model. The truth is that the Scrum approach, although not called Scrum at the time, was first presented as “the rugby approach” in 1986. In the January-February 1986 issue of the Harvard Business Review, Hirotaka Takeuchi and Ikujiro Nonaka described this approach for the first time.1 In their article, they argued that small cross-functional teams produced the best results from a historical viewpoint.
A real scrum
A scrum is formed in the field when eight players from each team, bound together in three rows for each team, close up with their opponents so that the heads of the front rows are interlocked. This creates a tunnel into which a scrum-half throws in the ball so that front-row players can compete for possession by hooking the ball with either of their feet.
Keep this definition in mind as we describe the development version of Scrum.
Ken Schwaber started using Scrum at his company in the early 1990s, but to be fair, Jeff Sutherland was the first to call it Scrum.4 Schwaber and Sutherland teamed up and presented this approach publicly in 1996 at Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA) in Austin, Texas. They collaborated to use their experience and industry best practices to refine the model until it achieved its current look. Schwaber described the model in Agile Software Development with Scrum in 2001.5
Let’s continue by looking at empirical process control and see what that means in software development.
Empirical Process Control
What is this model, or framework, all about? First, let’s define two ways to solve problems. We touched on the issues with projects in Chapter 1. When you have an issue that is similar time after time (like road construction, for example, or implementing a standard system), you pretty much know what to expect from the various tasks at hand. You can then easily use a process—the Waterfall model, perhaps—that produces acceptable-quality output over and over again.6 This approach is called defined process control .
- 1.
Transparency
- 2.
Inspection
- 3.
Adaptation
“Transparency means that the aspects of the process that affect the outcome must be visible to those controlling the process.”8 This means, to be able to approve the outcome, you must agree on the criteria for the outcome. Two people can’t say they’re “done” with a task unless they both agree on what the criteria are for “done.”
The next leg is inspection. The process must be inspected as frequently as necessary to find unacceptable variances in it. Because any inspection may lead to a need to make changes to the process itself, you also need to revise the inspections to fit the new process. To accomplish this, you need skilled inspectors who know what they’re inspecting.
The last leg is adaptation. An inspection may lead to a change in the process; this is one example of an adaptation. Another is that you must adapt the material being processed as a result of an inspection. All adaptations must be made as quickly as possible to minimize deviation later.
The code is reviewed against coding standards and industry best practices. Everyone involved in the review fully and mutually understands these standards and best practices. The code review occurs whenever someone feels that a section of code is complete. The most experienced developers review the code, and their comments and suggestions lead to the developer adjusting his or her code. 9
Simple, isn’t it? We could not have said it better ourselves.
Complexity in Projects
What makes a software development process so complex? We discussed this a bit previously, but let’s dive deeper here. In theory, building a software system may seem pretty straightforward. You write code that instructs the CPU to control the computer. How hard can it be? Alas, it isn’t that simple, we’re afraid. The people writing the code are complex machines in themselves. They have different backgrounds, IQs, EQs, views, attitudes, and so on. Their personal lives also add to their complexity.
The requirements may also be complex and tend to change over time. According to Schwaber, a large percentage of the requirements gathered at the beginning of a software project changes during the project—and 60% of the features you build are rarely or never used in the end. Many times in our projects, several people at the customer site are responsible for the requirements. Often, they have diverging agendas as to why and what to build. And just as often, the stakeholders have a hard time expressing what they really want. Only when they see a first prototype of the system do they fully begin to see the possibilities of the software, and only then can they begin to understand what they want.
Rarely is it the case that just one computer is involved in a system either. In general, there is interaction among several machines. You may have a web farm for your GUI, a cluster for your application tier, a back-end SQL Server, some external web services, and often a legacy system, all of which need to integrate to solve the needs of the new system.
Schwaber’s complexity graph
The projects in the anarchy area are chaotic and unworkable. To get them to their finish lines, you probably need to resolve serious issues before you even start the projects.
What Scrum tries to do is address this inherent complexity by implementing inspection, adaptation, and visibility, as you saw in the section “Empirical Process Control.” Scrum does so by having simple practices and rules.
What Scrum Is
The Scrum skeleton
During these inspections, team members evaluate each other’s work and the activities performed since the last inspection. If necessary, adjustments (adaptations) are found and implemented as quickly as possible. The iterations also conclude with inspections, when more adaptations can be made. This cycle repeats until it’s no longer funded.
All the requirements that are known at the beginning of the project are gathered in the product backlog, which is one of the artifacts of Scrum. We come back to this shortly. The project team reviews the backlog and selects which requirements should be included in the first iteration—or sprint, as it’s called in Scrum. These selected requirements are added to the sprint backlog, where they’re broken down into more detailed items (tasks). Many teams visualize their work in a sprint using a scrum board, which can be electronic or it can be a whiteboard with sticky notes. The board shows the tasks that have been determined for each backlog item and where in the development process each task is currently located (development, test, and so on).
The team then makes its best effort to turn the sprint backlog into a shippable increment of the final product. The team is self-managing, which means members decide collectively who does what and what the best way is to solve problems.
The increment is presented to the stakeholders at the end of the sprint so they can inspect it and, if necessary, the team makes adaptations necessary to the project based on stakeholder feedback . The sprint most often lasts 30 days, although, as mentioned earlier, we often see sprints that last two to four weeks. The length of the sprint depends on the sprint backlog items. When I took his Scrum master certification class, Ken Schwaber related that he once had a one-week sprint in a project. The reason was that the team malfunctioned, and this way he could identify the reason for this more easily and adjust the process so the project ran more smoothly.
The stakeholders’ adaptations and feedback are put into the product backlog and prioritized again. The team then starts the process over and selects the backlog items they think they can finish during the next sprint. These items are put into the sprint backlog for the next sprint and are broken down into more manageable items. And so it continues, until the stakeholders think they have received the business value they want, and then funding stops.
If you look again at the three legs of empirical process control, you can see that Scrum covers them nicely. Transparency is implemented by letting the team and stakeholders agree on the expected outcome of the project and of each iteration. Inspection occurs daily and also at the end of each sprint. Adaptations are the direct result of these inspections and are a necessary thing in Scrum.
Roles and Responsibilities in Scrum
- 1.
The product owner
- 2.
The Scrum master
- 3.
The development team
These people are, ideally, collocated to deliver potentially releasable product increments every sprint, but this can be hard to accomplish in a global organization (we’ll come back to this topic later). Together, these three roles form the Scrum team.
Product owners represent the business side of the organization. This role is very important to any team in an organization. Without clear and strong product owners, teams will probably soon run into problems.
Product owners are the voice of the customer, and they make sure the team adds value to the organization throughout the project (or product development). One of the most important responsibilities of product owners is to manage the backlog and make sure the backlog reflects the needs of the company. To do this, product owners need to work closely with the project/product stakeholders. They also need to work closely with the development team and the Scrum master.
Product Owner Responsibilities
Responsibilities | Meaning in practice |
|---|---|
Maximize the value of the product and the work of the development team. | Express product backlog items clearly. |
Manage the product backlog. | Order the items in the product backlog to achieve goals and missions effectively |
Be the business interface for high-level requirements. | Optimize the value of the work performed by the development team. |
Own and communicate the product vision. | Ensure the product backlog is visible, transparent, and clear to all, and make sure it shows what the Scrum team will work on next. |
Define the product road map. | Ensure the development team understands items in the product backlog to the level needed. |
Scrum Master Responsibilities
Responsibilities | Meaning in practice |
|---|---|
Ensure Scrum is understood and enacted. | Ensure the Scrum team adheres to Scrum theory, practices, and rules. Lead and couch the organization in its Scrum adoption. Plan Scrum implementations within the organization. |
Help those outside the Scrum team understand which of their interactions with the Scrum team are helpful and which aren’t. | Help everyone change their interactions to maximize the value created by the Scrum team. |
Remove impediments. | Protect the team by finding the resources to remove impediments so the team can work at a sustainable pace. |
Development Team Responsibilities
Responsibilities | Meaning in practice |
|---|---|
Do the work to complete the tasks by the end of each sprint. | Organize and manage the tasks assigned. |
The team is jointly responsible for fulfilling the forecasted work of the sprint. Although one name be written beside a task, all team members are responsible for completing that task. The goal of this team structure is to foster collaboration.
The development team is a self-organizing team, responsible for planning and executing what is on the sprint backlog. But, being self-organizing may be hard, so the Scrum master might help the team achieve this.
Development team members may have specialized skills and areas of expertise, but accountability for completing a product backlog task belongs to the entire development team. No one, not even the Scrum master, tells the development team how to turn the product backlog into increments of potentially releasable functionality.
Deliverables in Scrum
Now let’s take a look at the deliverables described in the Scrum guide.
The Product backlog
The product backlog is an ordered list of all requirements on the product. They could be business requirements or technical requirements. Nonfunctional requirements, such as response times, regulatory requirements, and so on, are often part of the “definition of done” the team maintains for its work, but they could also be included in the product backlog.
The product owner is responsible for maintaining the backlog but does so in close collaboration with other stakeholders and business representatives as well as with the development team. The product owner needs to prioritize the backlog so it reflects the needs of the organization.
You can think of the product backlog as a living document that follows the product for as long as the product lives. Items, or product backlog items (PBIs) are often estimated using story points or some other relative estimation technique, which is discussed later.
Sprint Backlog
The sprint backlog is the development team’s forecast of what it will deliver during the sprint. It is also the plan for how the team will do the work to accomplish the increment at the end of the sprint.
A burndown chart
This backlog is also a living document, but it is the development team that owns and updates the sprint backlog. Because it is a self-organizing team, team members can add and remove tasks as they wish as long as they follow the PBIs selected during sprint planning.
Product Increment
The increment is the sum of all backlog items completed during a sprint and includes what has been delivered so far in the project. At the end of a sprint, the new increment must be done, which means it must be in useable condition and meet the Scrum team’s definition of “done.”
Sprint Goal
The sprint goal is created at sprint planning and is the target for the sprint. It is a vision that guides the team to why they built the increment in the sprint.
Although the team and the product owner collaborate on this goal, it is important that product owners explain their vision for the coming three to four sprints.
Events in Scrum
Sprint planning
Daily Scrum
Sprint review
Sprint retrospective
Sprint planning
What can be delivered in the increment resulting from the upcoming sprint?
How will the work needed to deliver the increment be achieved?
This is the first meeting that occurs in a sprint, and the entire Scrum team is present. The meeting has a time box of eight hours for a calendar-month sprint. For a shorter sprint, this time is usually shorter. The Scrum Guide is not more specific than this, but a good rule is that the time box is four hours for a two-week sprint, six hours for a three-week sprint, and so on.
During the sprint planning meeting, any changes to the product backlog since last refinement meeting are discussed so that the team has a good product backlog from which to work. The capacity (or the number of hours available for the team to work during the sprint) of the team—for the entire team and for the individual competences in the team—is reviewed. After this is determined, the team starts to break down the PBIs, starting from the top.
The team does this by figuring out which tasks are needed for each PBI. The tasks could be designing, writing code, writing tests, performing tests, and so on. Then, these tasks are estimated with the remaining hours to build a good burndown chart during the sprint.
Daily Scrum
The daily Scrum event in Scrum
- 1.
What did you complete yesterday that contributed to the team meeting its sprint goal? This is a status report to the other team members.
- 2.
What do you plan to complete today to contribute to the team meeting its sprint goal? This is the plan for the next 24 hours.
- 3.
Do you see any impediments that could prevent you or the team from meeting its sprint goal? This is a way to identify any risks that could prevent the team from meeting its goal.
This is the suggested way of doing this meeting according to the Scrum Guide, but it is not set in stone. It is an efficient way for the team to get quick feedback and see if it needs to change anything in the short term.
Sprint Review
When the sprint comes to an end, it is time for the development team to show what it delivered and to get feedback on it from the product owner and stakeholders. This is done during a sprint review, which is another time boxed-meeting: three hours for a calendar-month sprint.
To this meeting, the Scrum team should invite persons that can give valuable feedback on the work delivered so far. To get this feedback, the team demonstrates real code (if software is the deliverable) to the audience and then discusses any feedback received. The feedback is then input to either the product backlog or to the retrospective following the review.
Sprint retrospective
During the sprint retrospective, the team reflects on the past sprint, and identifies and agrees on continuous process improvement actions.
Two main questions are asked during the sprint retrospective: What went well during the sprint? What could be improved during the next sprint?
The recommended duration is one and a half hours for a two-week sprint (and is proportional for other sprint durations).
This event is facilitated by the Scrum master.
There are many ways a retrospective can be performed. It is important not to let this meeting become boring. It is an important meeting, and the Scrum master needs to infuse inspiration into the meeting.
Backlog Refinement
The last event is the backlog refinement (previously known as backlog grooming). Backlog refinement is an ongoing process of reviewing PBIs and checking that they are prioritized appropriately and prepared in a way that makes them clear and executable for teams after they enter sprints via sprint planning.
Larger PBIs may be broken into multiple smaller ones; acceptance criteria may be clarified; and dependencies, investigation, and preparatory work may be identified and agreed as technical spikes (similar to a small proof of concept). Count on spending up to 10% of the available capacity on refinement.
Definition of Done
[Product owner, Sofia, stops by the office of developer Mike to check on how things are going.]
S: Hi! How’s the cool new feature you’re working on coming along?
M: It’s going great. I’m done with it right now and will start the next feature soon.
S: Great! Then I can show it to our customer, who’s coming here after lunch. He’ll be very excited!
M: No, no. Hold on. I am not “done” done with it. I still need to fix some test cases, do some refactoring, get it into the build process, and so on. I thought you were wondering if I had gotten somewhere with it.
For the most part, such discussions can be avoided if people sit down together at the beginning and write and sign a definition of done.
There are other reasons for having a definition of done as well. For the team to estimate a user story, the team members need to know when they’re done with it; otherwise, it’s very hard to complete the estimate. For a specific user story (requirement), you know it’s done when you’ve fulfilled its acceptance criteria. But where do all those general things such as style guides, code analysis, build automation, test automation, regulatory compliance, governance, nonfunctional requirements, and so on, fit in? They affect the estimate of a user story as well.
Here is where the definition of done comes into play. The definition of done tells you what requirements, in addition to the user story’s acceptance criteria, you need to fulfill to be done with the story. You include the general requirements in the definition of done because they affect all user stories in the end.
The definition of done is your primary quality document. If you don’t accomplish what is in it, you don’t deliver quality. It’s essential that the product owner and the team agree on the definition of done. The definition of done is part of the agreement between the team and the product owner.
There shouldn’t be an argument over this concept during the project. If the product owner thinks it’s too costly to use pair programming or TDD, have the product owner sign the definition of done, which specifies that these things have been removed. If, at the end of a sprint, the product owner complains about the number of bugs, you can present the document and say the product owner removed essential parts of the testing process, so bugs will be present.
Environments are prepared for release: First check that no unintegrated work in progress (WIP) has been left in any development or staging environment. Next, check that the continuous integration framework is verified and working, including regression tests and automated code reviews. The build engine ought to be configured to schedule a build on check-in. It may also trigger hourly or nightly builds. Also, check that all of the test data used to validate the features in the release have been validated.
Handover to support is complete: All design models and specifications, including user stories and tests, must be accepted by support personnel who will maintain the increment henceforth. They must also be satisfied that they are in control of the supporting environment. (Note: This may be elided in a DevOps context or when a development team follows the product through to support.)
Review is ready: Part of the work in a sprint includes preparing for the review. Sprint metrics should be available, including burndown or burnup charts. Any user stories that have not been completed need to be reestimated and returned to the product backlog.
Code is complete: Any and all to-do annotations are resolved, and the source code is commented to the satisfaction of the development team. Source code is refactored to make it understandable, maintainable, and better able to support future changes. (Note that the red-green-refactor pattern found in TDD is helpful here.)
Unit test cases are complete: Unit test cases are designed for all features in development and allow requirements to be traced to the code implementation, such as by clear, feature-relevant naming conventions. The degree of code coverage is established and meets or exceeds the standard required. The unit test cases are executed and the increment is proved to work as expected.
Peer reviews are complete: Source code is checked into the configuration management system with appropriate peer-reviewed comments added. The source code is merged with the main branch, and automatic deployment into elevated environments is verified. (Note: If pair programming is used, a separate peer review session might not be required.)
Testing is complete: Functional testing is done, including both automated testing and manual exploratory testing. A test report is generated. All outstanding defects (or incidents such as build issues) are elicited and resolved or accepted by the team as not being contraindicative to release. Regression testing is completed and the functionality provided in previous iterations is shown to work. Performance, security, and user acceptance testing is done, and the product works on all required platforms.
Story is clearly defined so that all team members can understand what needs to be done.
Story includes clear business value for product owner to prioritize.
Story includes acceptance criteria.
Story includes any required enabling specs, wire frames, storyboards, and so on.
Story is estimated and sized to complete easily within one sprint.
Story is free from external dependencies so the team can start working on it.
Let’s continue by looking at how you can manage requirements and estimations with an Agile mind-set.
Agile Requirements and Estimation
Agile requirements and estimation is a huge but important topic. This section covers some of the most important topics, but there are a lot of ways you can manage requirements and estimates. If you want to master this subject, there are several training courses you can take and books to read. A good starting point is to visit www.scrum.org or www.scrumalliance.com and see what they currently suggest.
Most of the Agile planning and estimation tips and tricks in this chapter come from the Agile community but aren’t specific to Scrum. Scrum really doesn’t tell you how to do specific things such as planning, estimation, and so on. Scrum is the process framework or process method you use for running your Agile projects. However, Scrum works very well with the concepts we look at next.
Requirements
As a <type of user>, I want <some functionality> so I may have <some business value>.
As a manager, I want my consultants to be able to submit expense reports through the Internet so that I can be more efficient in processing expense reports.
A user story implementation in the Scrum template Microsoft provides with TFS
User stories capture requirements at a high level and aren’t tangled up with detailed functions or implementation details. The details and nonfunctional requirements are instead captured as acceptance criteria for the user story. Based on these acceptance criteria, you can develop acceptance tests at the same time you write the requirements. The definition of done is also important here because it describes other critical requirements that all user stories need to fulfill before they’re considered done.
So, how can you begin gathering requirements before you start a project? Product owners should use any method they think is suitable. We often use story-writing workshops, during which key stakeholders, end users, business analysts, experienced developers, and others participate to brainstorm user stories. During such a workshop, you focus on the big picture and don’t dive into details. These big-picture user stories are often called epics because they’re large and not broken down yet.
But don’t you need to find all requirements at the beginning? No. And that is what makes Agile so great. The Agile concept builds on the fact that you acknowledge that you don’t know and can’t know all the requirements early during a project. New requirements and changes to early requirements pop up throughout the process, and that’s okay. The Agile approach takes care of this for you. You start with what you have initially and you continue handling requirements throughout the project. The short version is that you get started right away and are aware that changes and new requirements will come along.
When the initial requirements are finished, you have the embryo of the product backlog. However, before you can prioritize and estimate these user stories, you need to perform a risk assessment so you can get a grip on the risks associated with each and every one of them. A user story with a significant risk associated with it usually takes more effort to finish and should probably be done early during development.
Estimation
To know how much effort is involved with a user story, you need to estimate it. The sum of all initial estimates gives you a (very) rough estimate of how much time the entire project may take. But, because you know things usually change over time, you don’t consider this estimate written in stone.
You have what you need to estimate: You know the requirements, you have a definition of done, and you have acceptance criteria. In the Agile world, it’s recommended that you estimate time using a measure called story points. Story points aren’t an exact size—instead, they’re relative.
Here is an easy example we use when running Agile training. Take four animals—let’s say a cat, a pig, a zebra, and an elephant. Without being a zoologist, most people can say that the pig is three times the size of the cat, the zebra is twice the size of a pig, and the elephant is maybe four times the size of the zebra. If you have a couple of people sit down and discuss these animal sizes, you can pretty soon come up with an agreement about their relative sizes.
The same goes for user stories. Most developers can agree pretty quickly about the relative size of user stories. User story A is twice as big as user story B, and so on. You don’t need to be very experienced with the details of each user story to reach this agreement. Novice developers usually end up with the same estimates as experienced ones. Keep in mind that you aren’t talking exact time yet, only relative size.
The most common scale for expressing story points is a modified Fibonacci scale. This scale follows the sequence 1, 2, 3, 5, 8, 13, 20, 40, 100.
- 1.
The product owner/Scrum master reads the first user story.
- 2.
The team members briefly consider the user story and select a card each, without showing it to the others.
- 3.
The team members show their cards at the same time.
- 4.
If the result varies much, the people with the highest and lowest cards explain their reasoning.
- 5.
After a short discussion, the team plays again.
- 6.
When consensus is reached (or the team members are only one step apart), you’re finished.
- 7.
If the team still disagrees, you pick the highest value.
How long is the sprint?
How many working days are available in the sprint?
How many days does each team member work during the sprint? Consider planned vacation or other days off, planned meetings, and so on.
Deduct the time for sprint planning, review, and retrospective meetings.
The result is the capacity before drag (drag is waste time or unknown activities). You should measure drag in each sprint, but during initial planning it’s hard to know how much to include. The longer the project, the more accurate the drag. If you don’t know from experience what the drag is, 25 percent is a good landmark; included in this is the 10% backlog refinement.
- 1.
Estimate the first user story in detail.
- 2.
Break down what the team needs to do to deliver the story.
- 3.
Estimate hours for each activity, then summarize.
- 4.
Deduct the summary from the available time the team has in the sprint.
- 5.
Is there still time left? If so, take a new user story and repeat the process until no time is left.
- 6.
Summarize the number of story points from the stories included in the sprint.
Now you have a theoretical velocity.
At this point, you can make a rough time plan for the entire (at this point) project. This is good input for the product owner to use in discussions with stakeholders, and also for return on investment (ROI) calculations. The sprint planning process continues throughout the project, and the theoretical velocity can soon be replaced with one based on experience.
Backlog
A sample backlog in Azure DevOps
The product owner needs to keep the backlog in good shape throughout the project. This means it needs to be ordered. It also needs fine granularity at the top (perhaps three or four sprints down the list) and rougher granularity farther down. Keeping the backlog in your ALM tool set gives you the benefit of visibility and traceability. In the best of all worlds, you can link backlog items to code, check-ins, builds, and so on, giving you good traceability.
The product owner can also start to look at release planning at this point. It’s important to get an overview of coming releases, especially if you have a larger project. Release planning can be done on the epics (the larger user stories). A good approach is to look for themes among the user stories. What could be useful to release at the same time? If you find such features, you can make a theme from them and plan the theme for a certain release. When this is done, you can also do a very rough time estimate on the releases—and suddenly you also have a rough time plan for the entire project.
Now you have as much information that you could possibly ask for this early in a project. The next step is the sprint planning meeting, when the team members (as you saw earlier) select the backlog items they feel they can commit to during the sprint.
During the Sprint
During the sprint, you use several important meetings to inspect and adapt your process. We already covered the sprint planning meeting, which takes place at the start of each sprint, but there are other meetings as well, and all important to the Agile team.
Daily Standup
The daily standup is a meeting that takes place every day during a sprint. It is primarily a developer team meeting and is used to provide status updates to the team members. As the name suggests, this is a standup meeting, which comes from the practice of having attendees stand at a meeting because the discomfort of standing for long periods helps keep the meeting short.
The daily standups are kept short, at around 15 minutes, so participants can infer this isn’t a working meeting.
- 1.
What did you complete yesterday that contributed to the team meeting its sprint goal?
- 2.
What do you plan to complete today to contribute to the team meeting its sprint goal?
- 3.
Do you see any impediments that could prevent you or the team from meeting its sprint goal?
Although it may not be practical to limit all discussion to these three questions, the goal is to stick to them as closely as possible. If further discussions are needed, they should be scheduled for after the meeting. For instance, when team members ask for short clarifications and brief statements, they should try to remember that they should talk about those more after the meeting.
One of the important features of the daily standup is that it’s intended to be a communication meeting for team members and not a status update for management or other stakeholders. However, it can be valuable for product owners to participate in the meeting so they can identify any issues they need to address. This may remove the need for other status meetings afterward.
The meeting is usually held at the same time and place every workday. All team members are encouraged to attend, but meetings aren’t postponed if some team members aren’t present.
This practice also promotes closer working relationships with its frequency, need for follow-up conversations, and short duration, which in turn results in a greater rate of knowledge transfer—a much more active result than a typical status meeting.
Kanban
We’d like to present another method that is usually mentioned with Agile frameworks. Kanban is very popular in many organizations and is used by one of our customers today. Although our preferred project management method is Scrum for development projects, we realize that Scrum isn’t perfect in every situation. Scrum can be scary in the sense that it requires major changes in the way people work in their organizations. It can be hard to implement Scrum fully because humans seem to have an inherent resistance to change. And if you don’t have management with you, it’s even harder to implement. Wouldn’t it be great if you could find a process that was agile, but made it possible for you to make changes gradually?
Operations can also be difficult to perform using Scrum. Think about this situation for a moment. Let’s assume you have three-week sprints for your operations team. One week into a sprint, you suddenly realize there is a bug in the system that affects production. This bug needs to be fixed right away, so you write a backlog item and present it to the product owner. You need to bring it to the next sprint planning meeting, which will be held in two weeks. Then, it will take three weeks for the bug to be fixed, because you have three-week sprints. In the worst case, you’ll have to wait five weeks before the fix is available for deployment.
Of course, this is a rare situation. There are obviously ways to handle this better using Scrum. You could, for instance, always have a PBI of 10% of your available time set aside for bug fixes, and put this PBI at the top of your sprint backlog, allowing you to work on bugs as they’re discovered. But, we still don’t think Scrum is optimal for operations work. This is why we started to look at Kanban.
The name Kanban comes from the Japanese word for sign board. Kanban goes back to the early days of the Toyota production system. Between 1940 and 1950, Taiichi Ohno developed kanbans to control production between processes and to implement just-in-time manufacturing at Toyota manufacturing plants in Japan. The Kanban method was developed by David J. Anderson and is an approach to an incremental, evolutionary process as well as systems change for organizations.10 By using a work-in-progress limited pull system as the core mechanism, it exposes system operation (or process) problems. In such a pull system, tasks that are to be performed are pulled into the workflow, similar to when you pull a PBI into the sprint backlog. But, you can only pull a task into the workflow when there is free capacity to handle the task. It also stimulates collaboration to improve the system continuously.
The Kanban method has three basic principles11:
- 1.
Start with what you do now.
- 2.
Agree to pursue incremental, evolutionary change.
- 3.
Respect the current process, roles, responsibilities, and titles.
Let’s take a closer look at these.
Start with What You Do Now
The Kanban method doesn’t prescribe a specific set of roles or process steps. There is no such thing as the Kanban software development process or the Kanban project management method. The Kanban method starts with the roles and processes you have and stimulates continuous, incremental, and evolutionary changes to your system. This is the thing we like the best about Kanban. It allows you to continue using what you’ve invested in; the biggest difference is that you can implement big improvements to the existing process without worrying employees.
Agree to Pursue Incremental, Evolutionary Change
The organization (or team) must agree that continuous, incremental, and evolutionary change is the way to make system improvements and make them stick. Sweeping changes may seem more effective, but more often than not they fail because of resistance and fear in the organization. The Kanban method encourages continuous, small, incremental, and evolutionary changes to your current system.
Respect the Current Process, Roles, Responsibilities, and Titles
It’s likely that the organization currently has some elements that work acceptably and are worth preserving. You must seek to drive out fear to facilitate future change. By agreeing to respect current roles, responsibilities, and job titles, you eliminate initial fears. This enables you to gain broader support for your Kanban initiative. Presenting Kanban—rather than an alternative, more sweeping approach that could lead to changes in titles, roles, and responsibilities, and perhaps the wholesale removal of certain positions—may help individuals realize the benefits of this approach.
The Five Core Properties
- 1.
Visualize the workflow.
- 2.
Limit WIP.
- 3.
Manage flow.
- 4.
Make process policies explicit.
- 5.
Improve collaboratively (using models and the scientific method).
Let’s look at these and see what they mean.
Visualize the Workflow
The knowledge work of today hides its workflow in information systems. to understand how work works, so to speak, it’s important to visualize the flow of work. The right changes are harder to perform if you don’t understand the workflow. One common way to visualize workflow is by using a wall with cards and columns, called a Kanban board. The columns on the card wall represent the different states or steps in the workflow, and the cards are the feature, story, task, and/or result of the workflow, usually referred to as work items.
What is great is that you use the steps of your existing workflow. You don’t need to enforce a new way of working that changes the current approach dramatically. You basically place the Kanban process on top of what you have, then visualize this flow. This often feels more comfortable to coworkers and makes them more positive about the small changes you’re imposing on them.
A Kanban board in Azure DevOps
A Scrum board
Limit WIP
The WIP limit is important and tells you how many work items you can have in each step (column on the Kanban board). When the limit is reached, you can’t pull any new items into this step until a work item leaves the step. Limiting WIP implies that a pull system is implemented on parts or all of the workflow. The pull system acts as one of the main stimuli for continuous, incremental, and evolutionary changes to your system. It’s important, even critical, that WIP be limited at each stage in the workflow.
Manage Flow
The flow of work through each stage in the workflow should be monitored, measured, and reported. By managing the flow actively, continuous, incremental, and evolutionary changes to the system can be evaluated to determine whether they have positive or negative effects on the system.
A bottleneck in the workflow is discovered
Make Process Policies Explicit
It’s often hard to improve, or even start a discussion of improving, a process until the mechanism of the process is made explicit. Without an explicit understanding of how things work and how work is actually done, any discussion of problems often seems subjective or emotional. To get a more rational, empirical, and objective discussion of issues, explicit understanding is important. This will most likely lead to consensus on improvement suggestions.
Improve Collaboratively
Kanban encourages small, continuous, incremental, and evolutionary changes that stick. David Anderson also discovered this was very effective. Resistance to change, as we’ve mentioned, is easier to overcome if the steps are small and each step has a great payback. Teams that have a shared understanding of theories about work, workflow, process, and risk are more likely to be able to build a common understanding of a problem and thereby suggest improvement actions that can be agreed on by consensus. The Kanban method proposes a scientific approach to be used to implement continuous, incremental, and evolutionary changes, but the Kanban method doesn’t prescribe a specific scientific method to use.
Common Models Used to Understand Work in Kanban
The theory of constraints (the study of bottlenecks)
The system of profound knowledge (a study of variation and how it affects processes)
The lean economic model [based on the concepts of “waste” (or muda, muri, and mura)]
eXtreme Programming
eXtreme Programming (XP) is a deliberate and disciplined approach to software development. XP, like Scrum, was a direct outcome of the Agile Manifesto and incorporates many of its values. Aspects of these models were in the minds of their founders for a long time though and were used in many projects. XP stresses customer satisfaction—an important part of the Agile Manifesto. The methodology is designed to deliver the software the customer needs, when it’s needed. XP focuses on responding to changing customer requirements, even late in the life cycle, so that customer satisfaction (business value) is ensured.
XP also emphasizes teamwork. Managers, customers, and developers are all part of a team dedicated to delivering high-quality software. XP implements a simple and effective way to handle teamwork.
- 1.
Communication: It’s essential that XP programmers communicate with their customers and fellow programmers.
- 2.
Simplicity: The design should be simple and clean.
- 3.
Feedback: Feedback is supplied by testing the software from the first day of development. Testing is done by writing the unit tests before writing the code. This is called TDD, and it is becoming a frequently used practice in many projects, not only Agile ones. Later, we describe how TFS implements TDD.
- 4.
Courage: The software should be delivered to customers as early as possible, and a goal is to implement changes as suggested. XP stresses that developers should be able to respond courageously to changing requirements and technology based on this foundation.
RUP has use cases and XP has user stories. These serve the same purpose as use cases, but they aren’t the same. They’re used to create time estimates for the project and also replace bulky requirements documentation. The stakeholders (managers, end users, project sponsors, and so on) are responsible for writing the user stories, which should be about things the system needs to do for them. Stakeholders write stories because they’re the ones who know what functionality they need and desire; developers rarely have this kind of information. Each user story consists of about three sentences of text written by the stakeholder in the stakeholder’s own terminology, without any of the technical software jargon that a developer may use.
Another important issue is that XP stresses the importance of delivering working software in increments so the customer can give feedback as early as possible. By expecting that this will happen, developers are ready to implement changes.
The last topic we want to highlight with XP is pair programming. All code to be included in a production release is created by two people working together at a single computer. This approach increases software quality without impacting time to delivery. Although we’ve never had the benefit of trying this ourselves, coworkers with whom we’ve spoken who have used pair programming are confident that it adds as much functionality as two developers working separately. The difference is that quality is much greater. Laurie Williams of the University of Utah in Salt Lake City has shown that pair programmers are 15% slower than two independent individual programmers, but “error-free” code increases from 70% to 85%.12 In our opinion, this more than makes up for the decrease in speed.
We can make a reference to my old job as an assistant air-traffic controller here. Many are the times I sat in the tower when airplane traffic was so heavy that I needed help to keep track of every airplane. We’re aware that this isn’t the same thing, but the fact remains that two pairs of eyes see more than one pair—and this is what makes pair programming so attractive.
To learn more about XP, we encourage you to visit www.extremeprogramming.org/ .
Scaling Agile
What happens if you have more team members than can fit on a Scrum team (seven plus/minus two people)? What if 90 people are involved in the project? Can Scrum scale to handle this? According to Mike Cohn, in an article on the Scrum Alliance web site,13 you can use a process called Scrum of Scrums.
The Scrum-of-Scrums meeting is an important technique in scaling Scrum to large-team projects. These meetings allow clusters of teams to discuss their work, focusing especially on areas of overlap and integration. Imagine a perfectly balanced project comprising seven teams each with seven team members. Each of the seven teams would conduct (simultaneously or sequentially) its own daily Scrum meeting. Each team would then designate one person to attend a Scrum-of-Scrums meeting. The decision of who to send should belong to the team. Usually the person chosen is a technical contributor on the team—a programmer, tester, DBA, designer, and so on—rather than a product owner or Scrum master.
By using this technique, you can scale Scrum infinitely, at least in theory.
- 1.
Scrum of Scrums
- 2.
Scaled Agile Framework (SAFe)
- 3.
Scaled Professional Scrum (SPS)
Scrum of Scrums
The simplest and easiest way of scaling Scrum is by holding a Scrum-of-Scrums meeting. This way of syncing among teams is good when we have five to eight teams that need to sync.
Scrum of Scrums
The Scrum-of-Scrums meeting is like any of the other daily Scrums during which representatives talk about what their teams have done since the last meeting, what they are going to do to before the next meeting, and whether they have impediments that may affect their progress.
What has your team done since we last met?
What will your team do before we meet again?
Is anything slowing your team down or getting in its way?
Are you about to put something in another team’s way?
The reason for asking the last question is that it can be very helpful when we need to coordinate many teams.
Impediments are resolved with focus on the challenges of coordination among the teams. Solutions may include agreeing to interfaces among teams, negotiating responsibility boundaries, and so on.
The participants of the Scrum of Scrums tracks items via a backlog of its own, and each item contributes to improving among-team coordination. The Scrum of Scrums, however, very seldom manage that backlog like the team’s backlog with sprint planning. The reasoning is that the participants first and foremost are members of their own team and the focus is on that team.
SAFe
SAFe started out as a book by Dean Leffingwell in 2007. In 2011, it was named the Scaled Agile Framework (SAFe) and the organization Scaled Agile was founded. This is the organization that works on refining the framework. Right now, SAFe is in version 4.6.
SAFe is freely available on the company’s web site, so anybody can access the framework. There are numerous training sessions offered by SAFe partners and consultants, and it is strongly recommended that you attend such training before implementing SAFe. The Agile leaders in the implementation are the first that should be trained so that the implementation is as successful as possible.
When a team setup reaches more than 40 to 50 people, it might be worthwhile to look into SAFe because a Scrum-of-Scrums might not be effective any longer.
SAFe is available in four flavors at this point, and they all depend on how an organization is set up and the needs it has. There is no way to cover the entire framework here, so we look at these four versions as an overview in this chapter.
Essential SAFe
Essential SAFe
Cadence
A train starts with a two-day session called program iteration planning , or PI planning for short. It takes place during the last iteration of the previous ART. This is a big-room planning in which all members of all teams on the ART are present as well as business owners, product managers, and system architects/engineers. This meeting is the heart of SAFe and is necessary to run SAFe.
Establishes face-to-face communication across all team members and stakeholders.
Builds the social network on which the ART depends.
Aligns development to business goals with the business context, vision, and team and program PI objectives.
Identifies dependencies and fosters cross-team and cross-ART collaboration.
Provides the opportunity for “just the right amount” of architecture and Lean user experience guidance.
Matches demand to capacity, thus eliminating excess WIP.
Enables fast decision making.
- 1.
Committed PI objectives: A set of objectives created by each team with the business value assigned by the business owners
- 2.
Program board: A board that highlights new-feature delivery dates, feature dependencies among teams and with other ARTs, and relevant milestones
When the train leaves the station, the first iteration of the ART starts. The teams work according to Scrum and have the same iteration length. They can also use Kanban if they want.
Every team has one product owner, one Scrum master, and three to nine developers. This is the same setup found in the Scrum Guide.
The final iteration is called the innovation and planning iteration , which acts as an estimating buffer for meeting PI objectives and provides dedicated time for innovation, continuing education, PI planning, and inspect and adapt events. The goal of this iteration is to have some time devoted to innovation and not simply focus on deliver. During this iteration, the entire ART demos the combined solution and gets feedback on that.
The release train engineer (RTE) is the person who runs the train. Look on this role as an Agile PM. The RTE works in close collaboration with the system engineer/architect and the product management team.
There is a backlog, called the program backlog , for the whole ART managed by the product managers. The program backlog is the holding area for upcoming features, which are intended to address user needs and deliver business benefits for a single ART. It also contains the enabler features necessary to build the architectural runway (the technical infrastructure necessary for building features).
If we have bigger needs than Essential SAFe, we can scale up to the Large-Solution SAFe.
Large-Solution SAFe
Large-Solution SAFe
So, a Large-Solution SAFe is an ART (called a solution train) that runs more than one ART within itself. There is basically no upper limit on how many trains you can run in this configuration. Each individual ART is run as it is in Essential SAFe. What differs is that the setup is extended with a solution train engineer, solution architect/engineer, and solution management. Solution management works with a solution backlog the same way product management does for a program.
Portfolio SAFe
Portfolio SAFe
At this level you try to align your organization’s strategy to your portfolio execution by organizing the Lean–Agile enterprise around the flow of value through the value streams. Delivering the basic budgeting and necessary governance mechanisms, it ensures that investment in solutions provides the ROI the organization needs to meet its strategic objectives. With a large enterprise, there may be multiple SAFe portfolios, and the need for running full SAFe might arise.
Full SAFe
Full SAFe
Working with Portfolio SAFe and Full SAFe might require some major changes to the enterprise. You will most likely need to change your way of budgeting, for instance, because these two setups fund value streams instead of projects. But even for the other two SAFe setups, you can benefit from another way of funding programs and solutions. The goal here is that personnel in each value stream can make the decisions necessary without having a stricter and more time-consuming escalation process when a change is needed. You want to achieve a more decentralized decision-making process.
SAFe Implementation Road Map
SAFe Implementation Roadmap
- 1.
The need to make an obvious change: Making a change can sometimes be obvious. It could be that you have a burning platform, as SAFe calls it—meaning, you somehow realize that your way of working is not adequate any more. You have problems with delivering business value, for instance, and need to change.
- 2.
The need to make a change for future betterment: If you do not have a burning platform, sometimes proactive leaders will drive change so that you can be in a better spot in the future. Lean-Agile leaders must exhibit what Toyota calls “a constant sense of danger”—a never-ending sense of potential crisis that fuels continuous improvement. This might not be the most obvious reason to drive change, and leaders in this case must promote that maintaining the status quo is simply not a way forward.
With the implementation road map we get step-by-step guidance to drive a change regardless of the reason that kicked it off.
SAFe is one way to scale Scrum. Now let’s take a look at another way to do this. It originates from Ken Schwaber and Scrum.org .
NEXUS –SPS
In SPS, a Nexus ( nexusguide.org ) is an exoskeleton that rests on top of multiple Scrum teams when they are combined to create an integrated increment. Nexus is consistent with Scrum, and its parts will be familiar to those who have worked on Scrum projects. The difference is that more attention is paid to dependencies and interoperation among Scrum teams delivering one “done” integrated increment at least every sprint.
The result can be an effective development group of up to around 90 people. Scrum.org recommends three to nine Scrum teams with three to nine developers. This includes the Nexus integration team. For larger initiatives, the Nexus becomes a unit of scale. This is called Nexus+, a unification of more than one Nexus.
Roles: A new role, the Nexus integration team, exists to coordinate, coach, and supervise the application of Nexus and the operation of Scrum so the best outcomes are derived. The Nexus integration Team consists of the product owner, a Scrum master, and three to nine developers.
Artifacts : All Scrum teams use the same, single product backlog. As the PBIs are refined and made ready, indicators of which team will do the work inside a sprint are made visual. A new artifact, the Nexus sprint backlog, exists to raise this transparency during the sprint. All Scrum teams maintain their individual sprint backlogs.
Events: Events are appended to, placed around, or replace (in the case of the sprint review) regular Scrum events to augment them. As modified, they serve both the overall effort of all Scrum teams in the Nexus, and each individual team. Backlog refinement has become a proper event as well. This is an important practice for single teams, but at scale it becomes mandatory.
The Nexus framework ( nexusguide.org )
Refine the product backlog : The product backlog needs to be decomposed so that dependencies are identified and removed or minimized. PBIs are refined into thinly sliced pieces of functionality, and the team likely to do the work is identified as early as possible.
Nexus sprint planning : Appropriate representatives from each Scrum team meet to discuss and review the refined product backlog. They select PBIs for each team. Each Scrum team then plans its own sprint, interacting with other teams as appropriate. The outcome is a set of sprint goals that align with the overarching Nexus goal, each Scrum team’s sprint backlog, and a single Nexus sprint backlog. The Nexus sprint backlog makes the Scrum team’s selected PBIs, and any dependencies, transparent.
Development work : All teams develop software, frequently integrating their work into a common environment that can be tested to ensure the integration is done.
Nexus daily Scrum : Appropriate representatives from each Scrum development team meet daily to indicate whether any integration issues exist. If identified, this information is transferred back to each Scrum development team’s daily Scrum. Scrum development teams then use their daily Scrum to create a plan for the day, being sure to address the integration issues raised during the Nexus daily Scrum.
Nexus sprint review: All teams meet with the appropriate stakeholders to review the integrated increment. Stakeholder feedback may result in adjustments to the product backlog.
Nexus sprint retrospective: Appropriate representatives from each Scrum team meet to identify shared challenges. Then, each Scrum team holds individual sprint retrospectives. Appropriate representatives from each team meet again to discuss any actions needed based on shared challenges to provide bottom-up intelligence.
Let’s now take a look at Large-Scale Scrum (LeSS)
Large-Scale Scrum
- 1.
LeSS: Up to eight teams (of eight people each)
- 2.
LeSS Huge: Up to a few thousand people on one product
A single product backlog (because it’s for a product, not a team)
One definition of done for all teams
One potentially shippable product increment at the end of each sprint
One product owner
Many complete, cross-functional teams (with no single-specialist teams)
One sprint
The LeSS framework
In LeSS, all teams are in a common sprint to deliver a common shippable product, every sprint.
We can see that SAFe, Nexus, and LeSS take different approaches to scaling Agile. I would say that, so far, SAFe is the framework I see the most.
How Agile Maps to ALM
According to Forrester Research, Agile adoption has brought about significant support for ALM practices.14 The Agile way of working using frequent inspection and adaption coupled with an increased delivery cadence has made teams more disciplined in their way of working. When Agile was introduced, many people thought the opposite would be true, but reality has proved them wrong.
What parts of Agile map to the ALM process? Let’s look at what Forrester says.
Agile Captures Task-Based Work
Daily standup meetings allow team members to report progress against their tasks. As you’ve seen in this chapter, the PBIs are broken down into tasks during sprint planning, and each task is reported on, aggregating the results to the original PBI. Using digital tooling for this, such as Mylyn from Tasktop and TFS from Microsoft, allows you to capture effort, time, and other metadata, which can provide valuable insight into the real progress of your software development.
There Is Increased Frequency of Inspection
The iterative approach to development improves the frequency of inspection. During each sprint, you have project inspection at each daily standup. At the end of a sprint, during the sprint retrospective, you define what you’ve done well and what needs to be improved. This facilitates the feedback loop and, together with better visibility for reporting and traceability, has far-reaching implications for ALM.
Many Tools Collect Much Information
Continuous integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. 15
Test Artifacts Are Important
Agile teams often use TDD and increase the importance of test artifacts. Business analysts and quality assurance practices are converging, which is something that Agile methods encourage. Agile’s emphasis on the definition of done and frequent inspection increase the desire to link work items with test plans and cases. The result is that Agile teams create simpler requirements but see greater integration with test assets.
Agile Teams Plan Frequently
Agile teams plan more frequently than traditional teams. Planning takes place when creating work items, during sprint planning, during daily standups, during backlog refinement, and so on.
As a result, Agile teams have more information about their projects, such as estimates, actual results, and predictions. This enables ALM to move center stage, because planning activities are an important management component of an ALM application.
Summary
This chapter gave you an introduction to Agile concepts. It focused on Scrum and Kanban because they are so commonly used. You looked at the details of Scrum so you can understand why the Agile approach maps so well to the ALM process.
You also examined ways to scale Scrum and Agile to become more effective in larger projects or larger organizations. In addition, you learned that agile techniques can help you with visibility, traceability, and collaboration.