Creating Better Innovation Measurement Practices
Finding the right metrics to track innovation is by no means straightforward. To avoid common mistakes, executives should take a holistic perspective on their company’s innovation process.
For most companies, innovation is a top managerial priority. Many managers look at successful innovators such as Apple Inc. and Google Inc. with envy, wishing their companies could be half as innovative. To boost and benchmark innovation, managers often use quantitative performance indicators.1 Some of these indicators measure innovation as results or outcomes such as sales from new products. Others measure innovation as a process, using metrics such as the number of innovation projects in progress. And some track input measures such as the number of ideas generated, while still others focus on the innovation portfolio, by looking at factors such as the percentage of investments in breakthrough projects versus product line extensions.
Our research on innovation measurement suggests that the key managerial challenge is not identifying metrics — there is no shortage of measures to choose from. Nor should the goal be to find the perfect metric, since that quest is often futile. Rather, the crux of effective innovation measurement is to understand the problem that measurement should solve for the organization and, based on that insight, to design and implement a useful and usable innovation measurement framework appropriate to the organization’s needs.
In this process, identifying the right questions is usually more difficult than finding the appropriate answers. Executives need to understand the innovation challenges the company faces, how innovation is currently measured, and the extent to which current measurement practices help or hinder efforts to achieve the organization’s innovation goals. Only then will managers be able to steer clear of common innovation measurement mistakes.
Some of the most insidious mistakes involve placing too much value on data at the expense of meaning and getting bogged down with too many measures that provide contradictory advice and incentivize employees to do the wrong things. Although companies use performance measurement for almost any activity, measurement of innovation is by no means straightforward.
Managers often get hung up on selecting and implementing the appropriate measures. The goal of this article is to help managers ask the right questions about how to measure innovation and translate their insights into effective innovation measurement practices. We have developed a practical, step-by-step framework that helps managers identify whether their current innovation measurement practices need to change and, if so, how to go about measuring innovation more effectively. The framework is also aimed at companies that do not currently apply metrics systematically to innovation but would like to start.
Our framework is grounded in both innovation measurement literature and our research, which included a survey of managers as well as case studies of three global, innovation-intensive companies (a consumer goods company, a mining company, and a manufacturer of machinery). (See “About the Research.”) Our research allowed us to identify common mistakes, issues, and challenges associated with the measurement of innovation. Based on these insights, we developed our framework, which we verified and tested with additional interviews and workshops that included managers from more than 50 companies.
About the Research
In our research, which spanned more than three years, we explicitly sought to move beyond a focus on what aspects of the innovation process to measure to understanding whether companies’ current innovation measurement practices need to change and how to go about measuring innovation more effectively. Our data was collected in four phases.
First, we distributed a survey with open-ended questions (for example, managers were asked to write short essays) to 45 managers from 21 companies; all of the managers surveyed were involved in the measurement of innovation. We also conducted three expert interviews within a consulting firm that specializes in innovation management and measurement. Based on these inputs, we identified a set of mistakes, challenges, and issues pertaining to the measurement of innovation.
Second, we worked closely with three large companies: a consumer goods company, a mining company, and a machine products company. At those companies, we collected data from interviews, documents, meetings, and workshops. This generated a comprehensive analysis of the innovation measurement practices in these companies. Although the companies are active in different industries, we were able to identify commonalities in their practices. Based on data gathered in these first two steps, we developed a draft of our paper and process model.
Third, we conducted feedback interviews using our emergent findings. The interviewees confirmed to us that our general interpretations and conclusions were reasonable, while also providing additional insights that allowed us to substantiate and fine-tune our framework.
Fourth, we conducted five additional workshops, attended by more than 80 managers from more than 50 organizations. At these workshops, we described the traps and the process model and asked for feedback and verification. The workshops convinced us that our findings were relevant to a broad set of companies from a range of industries.
Three Common Traps
Most of the companies that took part in our research reported that they measure their innovation activities. However, they said their measurement efforts often failed to generate desired results. The root causes became clear in our initial analysis. For example, many companies used quantitative measures but neglected to use qualitative measures (such as employees’ skill level or the freedom employees have to explore fields outside the core business). Others overemphasized short-term measures over long-term measures, because they viewed short-term measures as more convenient. Some of the underlying issues, however, were less obvious2 and led to measurement activities that triggered unintended, unforeseen, unpleasant, or negative effects that were difficult to overcome or avoid.3 What emerged from our research were three important innovation measurement traps to avoid: (1) overestimating or underestimating the potential of innovation measurement, (2) measuring only the parts as opposed to the whole, and (3) overlooking the political power of innovation measures.
Trap 1: Overestimating or Underestimating What Innovation Measurement Can Do
Some of the companies we studied were too detailed in the way they measured innovation; they assumed that once things were measured they could be managed. Other companies did practically no measurement at all, on the assumption that measuring innovation was inherently counterproductive and harmful to creativity and novelty. Companies in the first group overestimated what innovation measurement could do; those in the second group underestimated it.
Comprehensive measurement of innovation allows managers to follow up on inputs, the innovation process itself, and its outcomes. However, excessively detailed measurement of everything harms innovation outcomes. As the saying goes, not everything that can be counted counts, and not everything that counts can be counted.
One company that overestimated what measurement can do was a global consumer goods company that does business in more than 100 countries. It added a wide array of innovation measures to ensure better management and optimize resource allocation. However, the measures it selected discouraged radical innovation, and the implementation resulted in information overflow, burdensome administration, and decision-making delays. Toward the end of our study, management realized it was on the wrong track and began to revise its innovation measurement practices.
Another company we studied that systematically underestimated the value of innovation measurement is consistently listed as one of the world’s top-five players in the mining industry. The company’s management was highly skeptical of innovation measurement, and as a result it hardly measured innovation at all. This made follow-up and management of innovation inherently difficult. At the end of our study, this company initiated a process of revamping its innovation measures and innovation-related activities.
Trap 2: Measuring Parts But Not the Whole
In the companies we studied, managerial attention was often directed to individual bits and pieces of the innovation process. Executives frequently failed to formulate a holistic overview of innovation inputs, activities, and outputs, or they focused too specifically on individual projects at the expense of their overall innovation portfolio.
The mining company, for example, made the mistake of frequently running similar projects in parallel. This prevented managers from focusing on the most pressing issues. It also disrupted the flow of the innovation process, created bottlenecks, and resulted in a growing frustration among both engineers and managers. What’s more, it harmed efforts to allocate innovation resources effectively. As one manager told us, “It is unclear who is actually entitled to start a new innovation project at our company and who has the overall responsibility for the portfolio of projects.”
In another example, a few years ago, when a large global manufacturer of heavy vehicles implemented measures to track the innovation process, front-load problem-solving, and ensure innovation projects were sufficiently resourced, it discovered that it had overlooked two important elements: measurement of innovation outputs and measurement of market preconditions for innovation. Although managers had a clear overview of what went into the process and how it proceeded, they could not follow up on innovation outputs or determine whether their innovation efforts were attuned with prevailing market conditions. In response, the company revised its innovation measures, adding new measures to reflect market conditions and how innovative projects were, plus a quarterly plan for when each project was expected to reach the market. The changes allow the company to better prioritize its innovation activities without diluting its overall innovative efforts.
Trap 3: Overlooking the Political Aspect of Innovation Measures
Any manager who wants to create or revise innovation measurement practices needs to understand the political implications of making changes. In most organizations, what gets measured is what gets done, and what gets done is what gets rewarded. Changing the way innovation gets measured therefore implies that some groups or goals will become more important while others will become less important. In the companies we studied, there were often heated discussions about what to measure, when, and why. Although such discussions can be fruitful, they can also be harmful, particularly if members of the organization remain unsure about what to focus on.
For example, at the consumer goods company mentioned earlier, the research and development (R&D) department’s innovation-related activities had historically been well-funded. Unfortunately, the investments did not result in sufficient insights about customer needs. As one manager commented, “The R&D department took ages to develop the ‘perfect product’ — but it wasn’t perfect because the market didn’t want it.”
The company attempted to make changes in measurement to overcome this problem but encountered fierce resistance, particularly from R&D. “It took about a year to get everyone on board and understanding our new ways of following up and tracking innovation,” one manager recalled. To overcome resistance and political struggles, management initiated a cross-functional initiative to look at innovation measurement.
Enhancing Innovation Measurement Practices
We encourage managers to recognize the three potential traps we have described before getting serious about creating new innovation measurement practices or revising existing ones. In our view, the biggest challenge is not identifying the right set of measures for a company but understanding the various blind spots that companies can encounter in determining how to measure innovation.
Mindful of the three traps, how can companies best implement new or revised innovation metrics? Drawing on our research, we have defined a three-phase process that can help companies improve their innovation management practices: (A) assess current innovation measurement practices, (B) improve core innovation measurement practices, and (C) deploy the improved innovation measurement practices. (See “A Framework for Improving Innovation Measurement.”)
A Framework for Improving Innovation Measurement
Companies can develop more effective innovation management practices in three phases: (1) assessing current innovation measurement practices, (2) improving those core innovation measurement practices, and (3) deploying the improved metrics.
Each phase involves distinct activities that companies should go through and includes questions to consider. The three phases unfold iteratively, where changes in one phase trigger adaptations in others. In the following section, we describe the three stages and the seven steps that comprise them, in detail. (See “Implementing the Innovation Measurement Framework.”)
Implementing the Innovation Measurement Framework
Companies that are changing their innovation measurement practices need to consider a number of key questions during each step of the process.
Phase A: Assess Current Innovation Measurement Practices
A key to successful change is to start by thoroughly mapping the current situation to assess current practices for measuring innovation. Carefully discussing and agreeing upon innovation measurement practices and the current innovation focus can enable more careful follow-up and prioritization of activities while avoiding dead ends and unproductive activities that produce data without meaning. This, in turn, helps executives avoid the first trap we identified in our research: measuring too much or too little.
Important questions to answer in this phase include:
- Do your current innovation measurement practices help or hurt your ability to achieve your innovation goals and priorities?
- To what extent are the current practices aligned with the overall company strategy?
Step 1: Identify existing innovation measurement practices. Most companies currently measure innovation performance somehow. But we found the usefulness of measures varies across companies. As one manager at a global consumer goods company told us, “We measure innovation, but we don’t have perfect key performance indicators.”
The first fundamental step toward improving innovation measurement is to identify and make explicit the number of innovation measures that are already in use. Putting them all on the table provides an overview and allows them to be categorized. The outcome is an overview of how innovation measurement practices fit with more general performance measurement practices in the company; how the company balances qualitative and quantitative measures of innovation; and how frequent, and in which areas, follow-up occurs on various activities that are driving innovation directly and indirectly. This approach to improving innovation measurement practices can be applied to all types of innovation (radical and incremental, service and technology, etc.).
Companies should also examine what innovation tools and methodologies are available and used, as well as existing routines for innovation measurement. These activities provide a thorough understanding of the baseline, or starting point, of a company’s innovation measurement practices. This knowledge is necessary to ensure that companies don’t change measurement practices based on faulty conclusions due to a lack of homework.
Step 2: Assess the current innovation focus and set priorities. Different types of innovation require different types of measurement. Therefore, companies trying to measure innovation must clarify their innovation focus and priorities. For example, radical innovation calls for a different measurement practice than more incremental forms of innovation. History is full of companies that developed a product that transformed the way business was done in a particular industry. However, the notion that innovation is solely about game-changing ideas, while dramatic, is misleading. Indeed, many companies have generated big returns from product line extensions and incremental innovations.
Once your strategic ambitions are set, it’s time to set priorities. Not everything can be accomplished at once, and prioritizing is an important aspect of innovation measurement. Here, companies need to understand that not all measures help with prioritization. Many results-based measures — for instance, those related to sales, profits, or customer loyalty — are based on lagging indicators. Such measures may be good for evaluating long-term effects, but they do not necessarily provide meaningful, short-term guidance. Specifically, lagging indicators don’t help executives prioritize which innovation activities they need to undertake, when, and in what order. Nor do they help determine which people to involve in a project, which issues need immediate attention, and which ones can wait.
It’s critical for companies to carefully work through, agree on, and articulate measures that help sustain a deliberate and explicit innovation focus with clear priorities. If not, they risk falling into both the first trap (measuring too much or too little) and the second (paying attention to the parts but missing the whole). A final point involves the importance of combining qualitative and quantitative measures. Many of the companies in our study relied on quantitative measures. However, some companies also used qualitative measures and found them to be an important complement to the quantitative measures in clarifying innovation focus and priorities.
Phase B: Improve Core Innovation Measurement Practices
Once companies have mapped their current innovation measurement practices and clarified their innovation focus and priorities, they can move on to updating existing measures or creating new ones. The focus of this second phase is to identify and develop suitable measures of innovation. In doing so, three broad categories should be considered: measures of the overall portfolio, measures of the innovation process, and measures of individual innovation projects.4 Together, these activities will help companies avoid the second trap we identified: measuring individual parts but not the whole.
In our research, many companies struggled with combining these different dimensions. One reason was that different stakeholders have different innovation foci. If the development of innovation measures fails to balance the interests of different internal groups, there is a clear risk (as indicated by the third trap) that the company will end up with a fragmented and inconsistent measurement system. This is not to say that purposeful and agreed-upon innovation measures are completely free from tensions and conflicts. As in all operations, there are trade-offs between different goals such as speed, resource utilization, and risk-taking; these trade-offs become painfully explicit when designing performance measures for the innovation portfolio, processes, and projects. But whichever measures a company selects need to advance the innovation strategy, with each measure providing clear answers to the questions of what is being measured, for whom, and why. This will help the company evaluate the usefulness of the measure and, later on, the assessment of goal achievement.
Important questions to answer in Phase B include:
- Do you have a balanced set of measures for your innovation portfolio, processes, and projects?
- Do you have the right number of measures?
- To what extent are the measures aligned with your strategy?
- To what extent do the measures contain potential conflicts?
Step 3: Develop or improve measures for evaluating the innovation portfolio. Companies increasingly apply portfolio thinking to their innovation priorities, seeking to balance radical innovation with incremental innovation, large projects with small projects, and high-risk innovation with low-risk innovation.5 In essence, portfolio thinking involves assessing innovation efforts as a group rather than individually. To accomplish this, existing measures may need to be adapted.
The global consumer goods company we studied provides a good example of how portfolio thinking works. Management developed a matrix to illustrate the difference between breakthrough projects, projects to create the next generation of an existing product, and cost-saving projects. The vertical axis reflected the technological or business-model change required (whether the project involved no change, evolutionary new changes, or radical new changes) while the horizontal axis represented customer perception of the degree of novelty in use. Overall, the matrix had 12 categories. The matrix enabled management to evaluate projects and assess whether the overall mix was in line with the company’s focus and priorities.
This company’s practice underscores the importance of thinking carefully about the portfolio balance. By working through and agreeing on the desired composition of the innovation portfolio, companies can clarify which projects are apt to be short-term wins and which ones are long-term and less certain. Doing so makes the link between innovation measurement and innovation strategy explicit.
Step 4: Develop or improve measures for evaluating the innovation process. Research on best practices for new product development has found that most companies have well-functioning processes for their innovation efforts. A formally documented new product development process6 has been the norm for many years.7 The models often feature five to seven overlapping stages, separated by predefined decision points for evaluating progress. These decision points have various deliverables and criteria to help managers make go/no-go decisions.
Where companies often run into problems, as we saw with the companies we surveyed, is with identifying a balance of measures for the innovation process. Some companies emphasized measuring the number of ideas generated or the size of the R&D budget, resulting in a fight for resources and attention later on as projects evolved and some stalled. Other companies focused too much on outputs and did not consider how long it took to go from an idea to a product in the marketplace.
In our view, companies should treat inputs, throughputs, and outputs with equal emphasis when developing or improving measures for evaluating the innovation process. Companies can go beyond measuring only the number of projects launched and also measure the speed in the innovation process (for example, performance against schedule, duration of the process, or average time to market). Additionally, when innovation projects stall, it is possible to monitor the amount of time a project has been static and the reasons for the lack of progress. Finally, companies do need to measure outputs (such as new product sales as a percentage of total sales, or profits from new product sales).
Step 5: Develop or improve measures for evaluating innovation projects. Most innovation activities are conducted through projects. In addition to developing innovation measures for the company’s portfolio and innovation processes in the aggregate, management needs to ensure that measurement also addresses the type and quantity of resources assigned to different innovation projects. In our study, two types of project measures emerged as particularly important. The first was the measurement of “slack” (the pool of resources available for producing a given level of organizational output).8 We found that better-performing companies ran fewer innovation projects but spent more time on each one, ultimately achieving successful projects at a lower cost.9 When there’s slack, key individuals for a project are less likely to be bottlenecks.10 Although many companies treat innovation projects as if they are predictable, innovation projects are in fact intrinsically uncertain and often include periods of slow progress due to lack of access to key individuals or resources.11 The ability to measure slack can counterbalance this uncertainty and lead to better innovation management.
The second type of measure was related to customer feedback and experimentation. This includes when and how often an innovation project interacts with customers and seeks their feedback on products, or when it might make sense to develop prototypes.
Phase C: Deploy the Improved Innovation Measurement Practices
In this final phase, companies need to implement and reinforce their chosen innovation measures to ensure they are actually being used, while discarding old measures that are no longer needed. Once companies reach this phase, they cannot afford to sit back and relax. As a rule, companies should be prepared to review and revise their innovation measurement practices regularly.
Step 6: Set routines for innovation measurement. A key activity is to set realistic targets for each of the measures identified in the prior phase. To enhance the likelihood of meeting the targets and obtaining internal commitment, management needs to designate owners for each measure.12 When setting targets, it’s also important to think specifically about measurement frequency (for example, whether it should be weekly, monthly, quarterly, or annually).
Step 7: Implement the new innovation measures and routines. Once targets are established, companies should provide training and follow up to make sure that people are properly using the new innovation measures. Managers need to determine specifically how to roll out the measures efficiently and to whom they should apply.
Finally, managers should develop a process for reevaluating the innovation measures and for examining the cause-and-effect relationships between measures and results. In addition to full-scale annual reviews, managers may want to make interim adjustments on a quarterly or semiannual basis to ensure that the measures are working as intended.13 In any case, executives should ask themselves how often and under what circumstances they will review their innovation measurement practices.
Why the Measurement Process Matters
In their efforts to become more innovative, companies are increasingly analyzing their innovation strategies, activities, processes, and projects. Evaluating and measuring innovation is central to these efforts. Although academics studying innovation measures previously downplayed the process through which measurement happens, we believe that approach has hindered the establishment of effective and efficient innovation measurement practices. Our framework is designed to help both individual executives and companies take control of their innovation measurement and understand the critical decisions, traps, and trade-offs involved — thereby allowing organizations to realize the full benefit of their innovation measurement efforts.
Anders Richtnér is an associate professor at the Stockholm School of Economics (SSE) as well as CEO of SSE Executive Education in Stockholm, Sweden. Anna Brattström is a postdoctoral fellow in the department of business administration at Lund University in Lund, Sweden. Johan Frishammar is a professor of entrepreneurship and innovation at Luleå University of Technology in Luleå, Sweden. Jennie Björk is a researcher and Mats Magnusson is a professor of product innovation engineering at KTH Royal Institute of Technology in Stockholm.
References
1. R. Adams, J. Bessant, and R. Phelps, “Innovation Management Measurement: A Review,” International Journal of Management Reviews 8, no. 1 (March 2006): 21-47; S.D. Anthony, M.W. Johnson, and J.V. Sinfield, “Institutionalizing Innovation,” MIT Sloan Management Review 49, no. 2 (winter 2008): 45-53; European Committee for Standardization, “Innovation Management — Part 1: Innovation Management System,” CEN/TS 16555-1 (Brussels, Belgium: 2013); OECD/Eurostat, “Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data,” 3rd edition (OECD Publishing, 2005); A.R. Shapiro, “Measuring Innovation: Beyond Revenue from New Products,” Research-Technology Management 49, no. 6 (2006): 42-51; and E. Mankin, “Measuring Innovation Performance,” Research-Technology Management 50, no. 6 (2007): 5-7.
2. J. Platt, “Social Traps,” American Psychologist 28, no. 8 (August 1973): 641-651.
3. R.M. Kanter, “Innovation: The Classic Traps,” Harvard Business Review 84, no. 11 (November 2006): 72-83; and L. Välikangas and M. Gibbert, “Boundary-Setting Strategies for Escaping Innovation Traps,” MIT Sloan Management Review 46, no. 3 (spring 2005): 58-65.
4. European Committee for Standardization, “Innovation Management — Part 1”; OECD/Eurostat, “Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data”; Shapiro, “Measuring Innovation”; and Mankin, “Measuring Innovation Performance.”
5. S.K. Markham and H. Lee, “Product Development and Management Association’s 2012 Comparative Performance Assessment Study,” Journal of Product Innovation Management 30, no. 3 (May 2013): 408-429; and R.G. Cooper, S.J. Edgett, and E.J. Kleinschmidt, “Benchmarking Best NPD Practices II,” Research-Technology Management 47, no. 3 (2004): 50-59.
6. G. Barczak, A. Griffin, and K.B. Kahn, “Perspective: Trends and Drivers of Success in NPD Practices: Results of the 2003 PDMA Best Practices Study,” Journal of Product Innovation Management 26, no. 1 (January 2009): 3-23.
7. Cooper, Edgett, and Kleinschmidt, “Benchmarking Best NPD Practices II”; and R.G. Cooper, “What’s Next?: After Stage-Gate,” Research-Technology Management 57, no. 1 (2014): 20-31.
8. N. Nohria and R. Gulati, “Is Slack Good or Bad for Innovation?” The Academy of Management Journal 39, no. 5 (October 1996): 1245-1264.
9. Markham and Lee, “Product Development and Management Association’s 2012 Comparative Performance Assessment Study.”
10. A. Richtnér, P. Åhlström, and K. Goffin, “‘Squeezing R&D’: A Study of Organizational Slack and Knowledge Creation in NPD, Using the SECI Model,” Journal of Product Innovation Management 31, no. 6 (November 2014): 1268-1290.
11. A. De Meyer, C.H. Loch, and M.T. Pich, “Managing Project Uncertainty: From Variation to Chaos,” MIT Sloan Management Review 43, no. 2 (winter 2002): 60-67; D. Reinertsen and L. Shaeffer, “Making R&D Lean,” Research-Technology Management 48, no. 4 (2005): 51-57; N. Modig and P. Åhlström, “This Is Lean: Resolving the Efficiency Paradox” (Halmstad, Sweden: Rheologica Publishing, 2012).
12. A. Papalexandris, G. Ioannou, G. Prastacos, and K.E. Soderquist, “An Integrated Methodology for Putting the Balanced Scorecard into Action,” European Management Journal 23, no. 2 (April 2005): 214-227.
13. Markham and Lee, “Product Development and Management Association’s 2012 Comparative Performance Assessment Study.”
Reprint 59101.
For ordering information, visit our FAQ page. Copyright © Massachusetts Institute of Technology, 2017. All rights reserved.