In This Chapter

Avoiding traditional and ineffective metrics

Making the most out of the available data

Optimizing the value of scrum

With scrum, metrics can be powerful tools for planning, inspecting, adapting, and understanding progress over time. Rates of success or failure can let a scrum team know whether it needs to make positive changes or keep up its good work. Time and cost numbers can highlight the benefits of agile projects and provide support for an organization’s financial activities. Metrics that quantify people’s satisfaction can help a scrum team identify areas for improvement with customers and with the team itself.

Double work agile is the practice of management expecting to see traditional status reports and meetings in addition to scrum artifacts, events, and appropriate metrics. This is one of the top pitfalls of scrum projects. Management is looking for one thing while teams are trying to do another. As a result, decisions are made based on the wrong information, teams burn out from doing double the work, and the benefits of scrum become minimized.

This chapter describes ten key metrics to help guide scrum project teams.

Sprint Goal Success Rates

One way to measure scrum project performance is with the rate of sprint success. The sprint may not need all the requirements and tasks in the sprint backlog to be complete to minimally realize the sprint goal. However, a successful sprint should have a working product increment that fulfills the sprint goal and meets the scrum team’s definition of done: developed, tested, integrated, and documented.

Throughout the project, the scrum team can track how frequently it succeeds in reaching the sprint goal and use success rates to see whether the team is maturing or needs to correct its course. Teams should always be stretching themselves, so 100 percent of the sprint backlog is not necessarily the goal. Making sure that individual requirements started get 100 percent “done” is important, but teams should always be stretching themselves, so success rates less than 100 percent sprint backlog completion should be considered an opportunity to learn and improve. Scrum masters should always be looking for ways to reduce drag on the team so that the team can set and accomplish higher and higher goals as they continue to accomplish more and more in each sprint. Sprint success rates are a useful launching point for inspection and adaptation.

Velocity is not a goal; it is a postsprint fact. An increasing velocity and an increasing sprint goal completion rate are both key indicators that a scrum team is continually improving efficiency.

You can find out more about setting sprint goals in Chapter 5, and reviewing them in Chapter 6.

Defects

To be truly agile, scrum teams need to implement agile practices like test-driven development (TDD) and continuous integration (CI). (See Chapter 12 for TDD and CI definitions.) Without quality practices like these, scrum teams will be ineffective at delivering quality as fast as the market demands because of the overhead of manual testing before each release and the amount of defects introduced that automation could easily catch.

It’s unlikely that any scrum team will be able to accomplish perfection in these areas, so any project is likely to have some defects. Agile techniques combined with the scrum framework help development teams proactively minimize defects.

Tracking defect metrics can let the development team know how well it’s preventing issues and when to refine its processes. To track defects, it helps to look at the following numbers:

• Defects during development: If the development team uses practices like automated testing and continuous integration, it can track the number of defects at the build level in each sprint.

  By understanding the number of build defects, the development team can know whether to adjust development processes and environmental factors.

• User acceptance testing (UAT) defects: The development team can track the number of defects that the product owner finds when accepting requirements in each sprint.By tracking UAT defects, the development team and the product owner can identify the need to refine processes for understanding requirements. The development team can also determine whether adjustments to automated testing tools are necessary.

• Release defects: The development team can track the number of defects that make it past the release to the marketplace.

  By tracking release defects, the development team and the product owner can know whether changes to the UAT process, automated testing, or the development process are necessary. Large numbers of defects at the release level can be indicative of bigger problems within the scrum team.

The number of defects and whether defects are increasing, decreasing, or staying the same are good metrics to spark discussions on project processes and development techniques at sprint retrospectives.

Time to Market

Time to market is the amount of time that a project takes to provide value by releasing working products and features to users. Organizations may perceive value in a couple of ways:

• When a product directly generates income, its value is the money it can make.
• When a product is for an organization’s internal use, its value will be the employees’ ability to use the product and will contain subjective factors based on what the product can do.

When measuring time to market, consider the following values:

• Measure the time from the project start until you first show value to the customer.
• Some scrum teams deploy new product features for use at the end of each sprint. For scrum teams with a release with every sprint, the time to market is simply the sprint length, measured in days.
• Other scrum teams plan releases after multiple sprints and deploy product features in groups. For scrum teams that use longer release times, the time to market is the number of days between the start of development of a feature and the next release.

Time to market helps organizations recognize and quantify the ongoing value of scrum projects. Time to market is especially important for companies with revenue-generating products, because it aids in budgeting throughout the year.

Return on Investment

Return on investment (ROI) is income generated by the product, less project costs: money in versus money out. On scrum projects, ROI is fundamentally different from ROI on traditional projects. Scrum projects have the potential to generate income with the very first release (which can potentially be as soon as the end of the first sprint) and can increase revenue with each new release.

To fully appreciate the difference between ROI on traditional and scrum projects, consider two scenarios with the same project costs that take the same amount of time to complete. I’m ignoring the additional documentation, meetings, and other expenses of a waterfall project and assuming that they could be kept at the scrum level to simplify the comparison. Both scenarios begin on January 1 and have the potential to generate $100,000 in income every month when all the requirements are finished:

• Scenario A is waterfall and releases when all requirements are finished on June 30 of the same year, and enjoys monthly revenue of $100,000 per month for each month thereafter through the end of the year (six months, $600,000 revenue).

• Scenario B begins releasing the highest-value and -risk features incrementally on January 31 after four one-week sprints, five months earlier than Project A. Monthly revenue is less as it builds up each month from the first release (that is, $50,000 in February, $60,000 in March, $70,000 in April, $80,000 in May, and $90,000 in June) until the entire project is complete.

  This extra revenue in each of the five months from February through June gives the project $950,000 in revenue for the year, $350,000 more than scenario A.

Like time to market, ROI metrics are a great way for an organization to appreciate the ongoing value of a scrum project. ROI metrics help justify projects from the start because companies may fund projects based on ROI potential. Organizations may track ROI for individual projects as well as for the organization as a whole.

Capital Redeployment

When the cost of future development is higher than the value of that future development, it’s time for the project to end.

The product owner prioritizes requirements, in part, by their ability to generate revenue. If only low-revenue requirements remain in the backlog, a project may need to end before the scrum team has used its entire budget. The organization may then use the remaining budget from the old project to start a new, more valuable project. The practice of moving a budget from one project to another is called capital redeployment.

To determine a project’s end, you need the following metrics:

• The value (V) of the remaining requirements in the product backlog
• The actual cost (AC) for the work to complete the requirements in the product backlog
• The opportunity cost (OC), or the value of having the scrum team work on a new project

When V < AC + OC (or AC + OC > V, as described in Chapter 5), the project can stop. The cost you will sink into the project will be more than the value you will receive from either continuing the project or starting the next project.

Capital redeployment allows an organization to spend efficiently on valuable product development and maximize the organization’s overall ROI.

Satisfaction Surveys

A scrum team’s highest priority is to satisfy the customer, both early and often, by delivering value. At the same time, the scrum team strives to motivate individual team members and promote sustainable development practices.

A scrum team can benefit from digging deeper into customer and team member experiences. One way to get measurable information about how well a scrum team is fulfilling its purpose is through satisfaction surveys:

• Customer satisfaction surveys measure the customer’s experience with the project, the process, and the scrum team.The scrum team may want to use customer surveys multiple times during a project, including at the very beginning to establish a benchmark for future comparisons. The scrum team can use customer survey results to examine processes, continue positive practices, and adjust behavior as necessary.

• Team satisfaction surveys measure the scrum team members’ experience with the organization, the work environment, processes, other project team members, and their work. Everyone on the scrum team can take team surveys.

  As with the customer survey, the scrum team may choose to give team surveys throughout a project. Scrum team members can use team survey results to regularly fine-tune and adjust personal and team behaviors. The scrum team can also use results to address organizational issues. Customer survey results over time can provide a quantitative look at how the scrum team is maturing as a team.

You can put together informal paper surveys or use one of the many online survey tools. Some companies even have survey software available through their human resources department.

Team Member Turnover

Scrum projects tend to have higher team member morale. One way of quantifying morale is by measuring turnover. You can look at the following metrics:

• Scrum team turnover: Low scrum team turnover can be one sign of a healthy team environment. High scrum team turnover (resulting from things like burnout, ineffective product owners who force development team commitments, personality incompatibilities, or a scrum master who fails to remove impediments, making the development team look bad in sprint reviews) can indicate problems with the project, the organization, the work, individual scrum team members, or overall team dynamics.

  Team members might be quitting, or managers might be stealing team members away to other projects. Either way, the result is increased thrashing and helps to expose organizational issues that should be addressed.

• Company turnover: High company turnover, even if it doesn’t include the scrum team, can impact morale and effectiveness. High company turnover can be a sign of problems within the organization. As a company adopts scrum, it should see turnover decrease.

When the scrum team knows turnover metrics and understands the reasons behind those metrics, it may be able to take actions to maintain morale and improve the work environment.

Project Attrition

Organizations with any size project portfolios should look at the rate of projects being cut short. Capital redeployment should not be confused with thrashing teams between projects at the whim of senior managers. Tracking project durations against capital redeployment analyses may expose trends of either ending projects prematurely or letting them run longer than needed, but most likely the former.

From these trends, portfolio managers can begin to look into reasons why they are getting cut short. High attrition may indicate such issues as thrashing, planning, prioritization, impediments, or cross-functionality.

For more on how scrum improves portfolio management, see Chapter 12.

Skill Versatility

Strong scrum teams are typically more cross-functional than weaker scrum teams. By eliminating single points of failure in a scrum team, you increase its ability to move faster and produce higher quality. Tracking skill versatility allows scrum teams and functional managers to gauge growth of cross-functionality. Chapter 13 talks about incentivizing and encouraging skill development for scrum teams.

When starting out, capture the existing skills and levels contained at each of the following organizational structures:

• Per-person skills and levels
• Per-team skills and levels
• Per-organization skills and levels

Over time, as each person increases the quantity and level of skills, each team and the organization will increase. It’s not about how many managers or directors you have by title in the organization that will deliver quality products and services to your customers. It’s about having team members who can all contribute to the sprint goal each day without the risk of single points of failure.

Manager:Creator Ratio

Typically, the larger the organization, the more likely a heavy middle layer of managers exists. Many organizations haven’t figured out how to function well without managers to handle personnel and training and development issues. However, you need to strike the right balance of managers and individuals who produce product.

Every dollar spent on someone who manages organizational processes is a dollar not spent on a product creator.

Track your manager:creator ratio to help you identify bloat and ways to minimize the investment that you’re making in people who don’t create product.