Benchmark Innovation Performance
How does your innovation performance compare to your competitors’?
It’s just human nature to be curious about how your performance compares with that of your competitors. But when it comes to innovation performance, this instinctive desire for benchmarking is also a valuable tool for innovation leaders. It can help leaders monitor the adequacy of their performance, communicate about ambition and achievement, and target performance improvement efforts.
Companies invest in innovation because it’s essential to them maintaining or enhancing their competitive position. That means relative innovation performance matters as much as absolute performance does. Benchmarking can be used to determine whether your relative performance is in line with your innovation strategy and objectives. Note that means you’re not using benchmarking to determine whether your performance is at the same level as your competitors. Your strategy should help you establish the relative level of performance you are targeting. For example, a longer average speed to market may make sense if you’ve decided to invest heavily in developing an emerging technology. A higher proportion invested in “core” makes sense if you’re following up on a recent breakthrough.
Benchmarking is also a valuable tool when communicating about innovation performance. Demonstrating the value you’re creating for the organization will typically carry more weight (especially when reporting “up” the organization) if you’re able to compare your performance against an objective industry benchmark. For example, reporting a vitality index1 metric of 30% is a lot more meaningful if you also report that the sector average is just 15%. Industry benchmarks of innovation performance can also provide succinct, compelling context for communicating targets “down” the organization.
Benchmarking of innovation performance doesn’t just have to be external. It can also be useful to compare performance within the organization. In fact, if external benchmarking helps you determine whether or not you need to improve innovation performance, then internal benchmarking is critical for determining where to improve innovation performance. It can, for example, help you determine which divisions or business units are most in need of improvement. Or which aspects of your innovation process to focus on.
Data availability makes benchmarking innovation performance a challenge
Unfortunately, meaningful benchmarking of innovation performance is challenging due to a lack of publicly available data.
There are very few innovation metrics that publicly traded firms share externally—particularly when it comes to output and outcome indicators. GAAP (& IFRS) require firms to report on research and development expenditure. This is at least a partial measure of inputs to the innovation process, even if its definition may be too narrow to cover all innovation related expenditure. See here for publicly available data on R&D expenditure collated by Commodore.
Patent data is generally available, but is a crude output activity indicator, at best. We place little value on patent activity data for benchmarking innovation performance for two main reasons:
- Not all innovations are patented (e.g. many are kept as trade secrets)
- The existence of a patent tells you virtually nothing about its value to a firm
Only the coarsest outcome indicators—revenue and earnings—have to be public reported by firms. And that also means relative performance indicators are rare—beyond those attempting to relate R&D expenditure to revenue2. McKinsey analysts thought they’d solved this problem when they proposed two conversion indicators: (1) new product sales per dollar of R&D expenditure and (2) gross margin per dollar of new product sales. Unfortunately, a critical component of these metrics, revenue from new product sales, is not nearly as available as the authors implied (although we note that these are still interesting metrics to look at internally). In fact, a quick analysis of earnings transcripts and SEC-filings from 2018 found just 40 examples of companies citing results for this metric (or something close to it)—see below to download this data. That compares with the ~2,000 US-listed firms that reported R&D expenditure in filings to the SEC for 2018.
Occasionally survey-based studies that benchmark innovation performance are conducted. For example, Stage Gate® International has previously published two benchmarking surveys of new product development metrics (in 2003 and in 2011). Unfortunately, such studies tend to be one-off or infrequent and give just a snapshot of performance at a point in time. This means they appeal to our curiosity, but rarely provide insight that is both actionable and timely.
Finally, niche services benchmarking innovation performance existing in some sectors. Clarivate and KMR Group are two examples in the pharmaceutical sector (where regulatory processes and investor pressure tend to mean there is more data available).
Develop a robust internal benchmarking program and “scrape together” what you can externally
So, what should you do to benchmark innovation performance? First up, we recommend ensuring you have a robust and systematic system of benchmarking internal innovation performance—to enable comparison of performance both across the organization and over time. Secondly, we suggest it’s worth at least taking a quick look at what data you can gather externally—it might be limited, but even that is better than nothing. The resources below are a good starting point.
1 – Percentage of revenue from new product sales.
2 – This has generally been a fruitless exercise. However, we recently came across an interesting article that appears to have had some success with an estimate of firms’ “ability” to convert R&D expenditure into sales growth. Mathematically: they performed a firm-level regression of the log of sales growth on the log of the lagged R&D expenditure to sales ratio, for five different lags (1 to 5 years); and averaged the five regression coefficients.
Most Companies’ Innovation Performance is Declining. Is Yours?
Research shows widespread, declining innovation performance.
At first glance, the idea that companies are getting worse at innovation seems ridiculous—“innovations” are such a prevalent part of our daily lives. And we are fed a constant stream of information about the “latest” innovations that will change our lives and disrupt our industries—AR/VR, artificial intelligence/machine learning/deep learning, the internet of things, blockchain, additive manufacturing, etc. But there is mounting evidence that innovation performance is declining—costs are increasing and returns are falling.
A study published by the National Bureau of Economic Research in 2017 found that 85% of a large sample of US public companies had seen a decline in research productivity between 1980 and 2010. The average change in research productivity across the whole sample? A 10% annual decline.
Anne Marie Knott, an Olin Business School professor, found that returns to companies’ R&D spending have declined 65% over the past three decades. Knott specializes in the measurement of firms’ innovation performance and is the developer of the research quotient measure of innovation performance.
Studies looking at more recent time periods suggest the trend hasn’t changed. Analysts at Accenture adapted the NBER methodology and found a 27% decline in the return on companies’ innovation spending in the last 5 years. And in a pharmaceutical industry-focused study, Deloitte analysts found that forecast returns on R&D fell to a 9-year low of just 1.9% in 2018. This in an industry with, arguably, some of the strongest drivers to innovate more efficiently (e.g. very high costs of R&D, pressure on drug prices from politicians and manufacturers of generic drugs).
You need to measure your own innovation performance.
The accumulating evidence is difficult to ignore. We believe it raises three urgent questions for innovation leaders:
- Are we getting worse at innovation?
- If so, how do we know what to fix?
- And, are our initiatives to fix these problems successful?
Clearly, being able to accurately and objectively answer the first question is of critical importance. If you don’t know whether your organization is getting better or worse at innovation, you’re essentially flying blind. That’s dangerous when your organization’s medium- to long-term competitiveness depends on your ability to innovate.
Answering the second question (what to fix?) is essential, because it ensures investment in your innovation process is carefully targeted. All too often we see firms who have failed to take a systematic, analytical approach to diagnosing what needs to be improved. Innovation leaders may rely too heavily on judgment (instead of testing judgment with data) and misdiagnose1. Or they essentially skip this question altogether and jump straight to adoption of the latest “solution” sweeping the world of innovation practice—think open innovation, design thinking, agile methods (NOTE: we have nothing against any of these approaches, provided they’re used in the right context and they demonstrably improve performance).
The final question (are our efforts succeeding?) is important because you need to check whether your interventions are working—and, if not, course correct. Often it’s necessary to put in place metrics that can provide an early indication of an intervention’s success. That’s because it can take too long for the impact to flow through to your ultimate measures of innovation performance (the ones you use to answer question 1).
If you can’t answer these questions, your system for measuring innovation performance needs to be improved.
A strong innovation performance measurement system is critical to answering all three of the questions outlined above. Try this simple test: when you’ve finished reading this post, can you immediately answer those three questions for your organization? Check the metric that measures efficiency2 (e.g. return on innovation investment)—is it trending up or down? If it’s trending down, can identify the weak spots in your innovation process, where data supports your judgment? This might require looking at efficiency indicators by division / business unit and by stage of your innovation process. Finally, for those initiatives you’ve introduced to improve innovation: do you have data to confirm they’re having a positive impact on performance?
If you’re unable to answer all of these questions quickly and easily, some aspect of your innovation performance measurement system is inadequate.
What next?
If you’re ready to start improving your organization’s system for measuring innovation performance, check out our post on the topic.
The Innovation Performance Measurement System Check-up
Problem: Most firms neglect to evaluate the effectiveness of their innovation performance measurement systems.
Action: Conduct a quick check-up by following the 3 steps below.
High-performing companies know that using metrics effectively is critical to managing innovation. And they see the payback—they attribute 20% of their improvement in innovation performance to better management by metrics.1 And yet, research conducted by the Innovation Research Interchange indicates most organizations don’t review the effectiveness of their measurement systems, even on an ad-hoc basis. If your organization is among those, follow the 3 steps below.
Step 1. Check your innovation performance measurement system’s intended functions.
Over time, innovation leaders can find themselves with a hodgepodge of metrics—some that they collect because “it’s what we’ve always done;” some because someone read that a competitor collects them; some because the CFO asked for them; etc. But then those metrics don’t end up being used to inform specific decisions. In the best case, reporting unnecessary metrics is a waste of time and resources. In the worst case, those metrics end up sending contradictory or confusing signals that lead to bad decision making.
It’s important to be deliberate in identifying how the information generated by your innovation performance measurement system will be used. The diagram below lists the possible functions. First, choose the functions your system is intended to accomplish. Next, identify “orphan metrics”—metrics you track that don’t correspond to any of those functions.
Last, determine whether your system is effective at each of the intended functions. The following, corresponding questions may help.
- Do you have the information you and leadership need, to know if your portfolio(s) and mix align with your firm’s strategy, risk tolerance, and innovation ambitions?
- Do you have the information you need to determine portfolio-level investments? What about to allocate resources to projects (e.g., which new projects to fund, which existing projects to continue and which to kill)?
- Can you quickly ascertain whether projects are making sufficient progress toward unlocking future value?
- Using information from your measurement system, can you communicate with leadership, innovation teams, and other functions about innovation priorities and progress?
- Can you tell where improvements in innovation capabilities are most needed?
- Do you have the information you need to evaluate team performance?
Step 2: Ask your stakeholders whether the innovation performance measurement system is working for them.
We’re big believers in getting to know the customers of your innovation performance measurement system, just as you would for a new product or service concept. So, to review the system’s effectiveness, ask your stakeholders for input. Choose 8 to 10 people including representatives of your team, leadership, and other internal “customers” for innovation. Schedule short (<30-minute) conversations with each.
Remind each stakeholder of the functions of your measurement system, as identified in Step 1. For example: “We want to make sure we stay aligned with strategy, get more systematic at allocating resources, and keep everyone posted on the progress being made. We’re having these conversations to make sure we’re accomplishing those things and meeting your information needs.” Then proceed with a few focused questions like:
- Of the information we report, what is most helpful or interesting to you? What else would you like to know, or learn about?
- Does our reporting schedule/cadence align with decisions you need to make?
- What concerns do you have about the state of the innovation function? Is there anything you believe we should be actively encouraging (e.g., more risk-taking) or discouraging (e.g., short-termism)?
Step 3: Eliminate unnecessary metrics and prioritize improvements.
Next we recommend the following steps to identify specific actions to take:
A. Eliminate the “orphan metrics” you identified in Step 1.
B. Make a “long list” of the areas in which your system needs improvement, drawing from the questions you answered “no” to in Step 1; and the input from stakeholders on preferences, cadence, and general concerns.
C. Select the 3-5 areas above that, if improved, would have the most appreciable positive effect.
D. Translate those 3-5 areas into actions to improve, and tackle those first.
This will get you started in addressing some common problems with innovation performance measurement. Be on the lookout for resources for more advanced fine-tuning in the coming weeks. If you want help sooner, email Phil. We’d be glad to review your system against best (and emerging) practices and provide some recommendations for improvement.
1 RTEC (2010). Driving effective R&D performance through effective measurement.
2 Adapted from: Godener, A. and Söderquist, K. E. (2004). Use and impact of performance measurement results in R&D and NPD. R&D Management 34.
The Operational and Creative Mindsets in Innovation Portfolio Management
Problem: Innovation leaders need to balance Operational and Creative mindsets, but it’s easy to overemphasize one.
Action: Answer 10 questions to review and calibrate your practices and metrics.
As an innovation leader, you have to enable creativity and exploration while ensuring rigor and discipline—you need both the Creative and Operational mindsets.
The Operational mindset seeks constant improvement. People who lead with the Operational mindset emphasize efficiency and want to minimize variability. They’re comfortable with quantitative data and can break any problem down into its component parts.
The Creative mindset seeks constant discovery. People who lead with the Creative mindset thrive amidst change and the unexpected. They’re comfortable with qualitative data and make connections where others do not.
Strengths of the Two Mindsets
We all have a natural leaning, but overemphasizing one mindset can create problems.
Neither mindset is inherently better for an innovation role, and there is lots of potential overlap. The Creative values quality and discipline, and the Operational values ingenuity and problem solving. But just like having a weak side of your body can cause pain or injury, overemphasizing one mindset can hurt innovation efforts. We often find clues to the dominant mindset in innovation measurement systems.
Leaders and organizations overemphasizing the Operational mindset may be less flexible. They might resist the changes in approach or metrics that teams need over the course of front-end projects (those in which you’re figuring out what to make for whom). They might measure adherence to budget and schedule but not learning, the most important output of any innovation project. They may view the primary function of their measurement system as monitoring but neglect its function for communication and buy-in.
Those that overemphasize the Creative mindset may not have metrics at all. If they do, they likely favor measuring activity related to innovation capabilities (think # employees trained) over innovation performance. They might go through the motions of tracking and reporting progress, but don’t really change behavior based on their metrics. And they, too, struggle with communication. People overemphasizing Creative may not translate their results into the first language of different stakeholders, like the C-suite or certain business unit leaders.
So balance is key. As is the case with the body, you have to make an extra effort on the weak side to even things out. As someone who leads with Creative mindset, the creative comes more naturally to me, so I have to work a little harder on the operational. I spend more time reading about quantitative analytics. I set objectives and key results for my exploratory work. I seek out collaborators who lead with the Operational mindset.
Action: Review and calibrate your practices and metrics.
Not sure which of the mindsets you’re leading with? Consider the following questions.
- Do you allow for changes in approach and metrics, as teams learn or strategy evolves?
- Do you measure learning?
- Did you involve your team and other stakeholders in the selection of key metrics?
- Do you treat cycle time (e.g., idea to market) differently for front-end projects?
- Do you have too many metrics? (check: can you articulate what decision is made based on every metric you track?)
You (or your organization) might be overemphasizing the Operational mindset if you answered “no” to any of those.
Now consider these:
- Do you measure innovation performance?
- Do your metrics track inputs, outputs, and outcomes?
- Do you track alignment with strategy?
- Are your teams really using your metrics? (check: can they tell you what behavior changes based on each metric you track?)
- Are you able to report your metrics in ways that are meaningful to multiple audiences (e.g., your team, BU leaders, and the CFO)?
If you answered no to any of those, you might be overemphasizing the Creative mindset.
To calibrate, review your “no” answers and identify what you’d change to get to “yes.” And look for more resources on balance from us in the coming weeks. If you want help sooner, email me. We’d be glad to do a quick review of your current practice/metrics and provide some recommendations for better balance given your goals and portfolio mix.
The Classic Mismatch: Expectations for Innovation from Different Levels in the Organization
Business leaders and consultants have been talking about the importance of working across functions to improve innovation performance, for decades. But lack of alignment across levels of the organization is just as problematic for innovation, if not more so. These disconnects are not caused by conflicting expertise or business unit priorities, but by entirely different expectations for innovation. If you’re responsible for growth from innovation, chances are you’re stuck in the middle of this classic clash.
The CEO talks about the need for transformational innovation and sets ambitious growth targets. But she expects the linear progress, quick results, and certainty she gets from other functions.
The Innovation Leader hears those ambitious targets and expects the CEO to be prepared for the exploration and risk-taking they imply. He expects The Team to follow suit—to embrace uncertainty and formulate new processes when the old ones are too restrictive.
The Team is a product of the broader culture of your organization. In the absence of a clear message and incentives to the contrary, they’ll pick up on what the CEO typically values and do exactly that.
There are, of course, exceptions to where these typical perspectives live in an organization—maybe your leadership are the creative visionary types, and it falls to you to manage risk. Regardless, the problem lies in the tension between expectations and how that tension manifests.
Different expectations for innovation at different levels can result in constant battles.
Without better alignment of expectations across levels, you can expect ongoing battles about:
- Personnel. Who are the right people for innovation projects? Do they work here already or do we need to look outside? What’s the right mix of skills on a given team? What are the right goals and incentives?
- Process. What sort of methods should we use? What happens first? What are the gates?
- Time Horizons. What’s a reasonable timeframe for a given project? When should we expect to hear recommendations? What’s the time to market?
- Results. What does success look like? What’s the criteria for approval? How will my performance be measured?
You’d answer every one of those questions differently, depending on which of the above perspectives you have. NB: these conflicts will be exacerbated for front-end projects (because they’re the least certain) and potentially disruptive portfolios (because they feel the most threatening).
Conversations about the right metrics can act as a forcing function for alignment.
You might think resolving these sorts of differences require trainings or long philosophical debates, but there’s a more straightforward solution. Talk about metrics with your leadership and team, early and often. I say “early” because it’s important to set expectations at the outset. Innovation managers sometimes put off establishing metrics, saying “we’ve just got to get started!” But if stakeholders are asking about measures of progress and impact, you’ve waited too long.
As someone charged with driving growth through innovation, you’re the best person to lead that conversation. It’s not enough to say that innovation, broadly, or a transformational portfolio should be treated differently. Instead, explain what’s different about your work or context, and the risks of applying traditional metrics. Be specific about the types of metrics that should be used instead. Those will depend on your unique context, but you may want to emphasize measures of learning (e.g., those generated by rating scales or innovation accounting) and de-emphasize time and predictability metrics. We help clients gain this internal alignment in a lunch and learn that covers the basics of innovation measurement, common pitfalls, and how to choose the right metrics for a given portfolio type/mix.
None of this is to say you can design a measurement system in a single meeting. But, using metrics to ground the conversation, you can get on the same page with your leadership and team faster than you might think.
If you want to get better aligned with your leadership and/or team,
- Download our Guide to Measuring Your Innovation Portfolio. Among many other topics, it covers understanding your stakeholders (page 38), measuring learning in the innovation context (pages 18-19), and measuring uncertain outcomes (20-22).
- Email me: I’m happy to have a quick call about how to frame and manage a conversation to align expectations.
Measurement for breakthrough projects – Practitioner’s Dispatch Series
It’s always helpful to hear from people who have struggled with and overcome a problem you’re facing. With this post, we’re excited to launch the Practitioner’s Dispatch, a new series featuring innovation practitioners’ battle-tested advice on common problems. First up is Stewart Witzeman, who has more than 30 years of experience in innovation management. We asked him to give you his best advice on measurement for breakthrough (see also: disruptive, Horizon 3, etc.) projects. His bio follows the post. Thanks, Stewart!