Measure What You Need

The way companies typically approach deciding what to measure is through analyzing the skills and qualities that specific roles, teams, or business units need. This involves defining role requirements or what the company needs people to do. Sometimes this is achieved through a formal process like a job analysis, and in some countries having a structured job description is actually a legal requirement. Other times and in other countries, requirements are defined through more informal or intuitive means. Yet it is always there to some degree, even if it is just an idea in the hiring manager's mind. There is a picture or list, then, of what you need and are looking for.

What is important is to make sure that this list is explicitly stated and clearly distinguishes the two or three things that are most critical for ensuring that people succeed. Then, wherever possible, ensure that your choice of measures is led by this list of most critical qualities or competencies by the type of talent the business needs. As principles go, it may sound obvious, but it is too often overlooked.

Ask About Validity

Sometimes, of course, there is only a vague idea of what a role or the business needs, and other times the list of what is required is just too long. When this happens and there are thus no clear requirements for guiding the decision of what to measure, businesses tend to revert to what they know or feel familiar with. This is a bad idea.

As we have seen, there is no one measure that you should always use: different jobs require different qualities. So if you always use the same set of measures and tests for all roles, the chances are that sometimes they will not help you much, and they may even be misleading.

Instead of reverting to the familiar, then, when choosing what to measure, the one question all businesses should always ask is this: “How predictive of success is this factor in this particular type of role?” or, in other words, “How valid is this measure?” Because validity figures can tell you whether a measure predicts performance, checking validity is a way of checking that you are genuinely measuring what you need.

We have encountered many leaders who do not seem to be in the slightest bit interested in validity, which we frankly find amazing. If you are going to pay good money for measurement results, then you need to make sure that the information they are providing is accurate and relevant. On a purely commercial level, anything else is just bad business.

Validity is a technical subject and can be complicated. So in chapter 5, we present a list of specific questions that you can ask about the validity of measures to help you decide whether they can help you predict success. For the moment, though, we just want to highlight the basic rule that you should always ask about validity. In other words, be led by the facts and science, not by traditions or familiarity.

Ask About Incremental Validity

Asking about validity can help you understand which measures are the most predictive of success. Yet as we have noted, talent is made up of a mix of multiple qualities and abilities, so you need to use multiple measures, and to work out which combination is best, you need to ask a different question. Just asking about validities will not work. This is because when you ask how valid a particular test is, you are asking how good a measure it is on its own, separate from anything else.

To work out what the best combination is requires a different question: one about incremental validity. This is the amount of validity that one measure has over and above another one: how much additional information or validity it provides with whatever other measures you are using. For example, we know that personality-based integrity tests are nowhere near as good as intelligence measures when it comes to predicting success. On this basis, we might decide not to use them. But when we look at the incremental validity they offer over intelligence tests, it is a very different story. We find that they can add around 0.14 validity points to the 0.5 or so validity figure that intelligence measures give us.¹ So we now have total validities of 0.64, which are able to account for over 40 percent of the causes of success. Likewise, personality tests appear to offer some decent incremental validity over intelligence with most roles, though precisely how much depends on the role and the tests used.²

Sometimes, of course, measures that you might expect to offer incremental validity do not. There is evidence, for instance, that if you already use both intelligence and personality tests, adding a measure of emotional intelligence will offer little extra validity.³ Another example comes from the combination of intelligence tests with work sample tests. For medium-complexity jobs, the validity of intelligence tests is about 0.51, and for work sample tests, it is around 0.54. On this basis, you might think that by combining the two, you would get a superb predictor of success. Unfortunately, this is not the case.

Research shows that performance on work sample tests is largely a consequence of intelligence: intelligent people tend to do better on them. In other words, a large part of what work sample tests measure is intelligence. As a result, they offer little incremental validity over intelligence tests.⁴ It might look as if they are measuring different things, but by asking about incremental validity, you can see that they are assessing more or less the same thing.

Incremental validity is thus the reason that using more measures is not always better. We recently met representatives of one company that used seven different psychometric tests in its recruitment processes in addition to interviews. They simply assumed that the more tests they had, the more able they would be to predict performance. Yet had they considered the incremental validity that each offered over the others, they would have found that this is not the case.

So while it might be tempting to use as many as possible of whichever tests have the highest validities, focusing only on validity is likely to lead to the wrong combinations or at least not the best ones. Asking about incremental validity can make all the difference: it is the yardstick for gauging which combinations of factors are the most predictive of success.

Since different jobs require different qualities, the most effective combination will vary between roles and companies. Yet as a general guide, we recommend you begin by identifying a foundation for your measurement process, such as a specific competency or job criterion. This is typically the most important thing that the business in general or a role in particular needs from the individuals being measured. You can then look to see which among the measures that you could use and are relevant to the role offer you the best incremental validity over your starting point.

And the next time a vendor shows you a measure, do not be satisfied just with knowing what its validity is. Ask what level of incremental validity it can offer over the other factors you are measuring. Because little research has been conducted on this issue so far, chances are that they may not know the answer for the specific tests you are considering. Nevertheless, they should have a general idea, and asking the question will focus both you and them on the issue and thereby help ensure that you get effective combinations of measures and do not employ redundant tests.

The Problem with Potential

All of this sounds good, and many of the potential models used are touted as being highly able to predict success. Some vendors even suggest they can do this many years in advance with validity levels exceeding 0.70. This would mean that they can account for around 50 percent of the reasons people succeed or not. However, few of the models and tools that abound have been independently validated, and there is a general lack of evidence about which factors can best predict long-term success. From the research that has been done, though, two broad conclusions can be drawn about which factors to measure when gauging long-term potential.

First, the factors that are most able to predict long-term success are generally the same as those that best predict current performance. If you look at some specific roles, this might not be true. The factors that predict current performance in information technology engineers, for example, may be slightly different from those that predict which engineers will reach the top of the profession. Technical expertise, for one, may be more important for current performance. But if you look overall at the general indicators of long-term potential across all roles, they are pretty similar to what predicts short-term performance.

Second, there is no conclusive evidence that long-term success can be predicted with much accuracy by any model or single pattern of characteristics. Looking at particular combinations of personal qualities can, of course, be better than merely relying on random chance to identify those with high potential. But let's not kid ourselves. Given the sheer number of variables involved in determining success over long periods of time, identifying potential is very difficult. From a purely technical viewpoint, trying to accurately predict potential is like trying to shoot an ant from a mile away. At night.

How to Predict Long-Term Success

So what should you do if you need to try to measure potential? After all, many businesses rely on these ratings to guide developmental investment. In general, the three principles we have described for choosing what to measure apply just as much to when you are trying to measure long-term potential as to when you are trying to predict short-term success. You must ask what type of talent the business needs. Which of these abilities and qualities are most able to predict success? And which combinations of factors give the highest validities? To help decide how to measure potential, though, there are a few extra guidelines we would add:

• Do not feel that you need to use a specialist measure of potential. By all means, supplement your evaluations of performance with other measures, but they do not need to be specialist models of potential. Remember: depending on the role, a combination of intelligence and personality tests can predict success with a total validity of between 0.55 and 0.65. A specialist measure of potential will need to match or better this.
• Do not look too far ahead. The further ahead you try to predict success, the less likely it is that you will be right. To have a decent chance of accurately predicting which people are most likely to be successful, focus on just trying to identify who has the potential to be promoted to the next level. This will make it easier to define what abilities and qualities are most required to succeed and improve your chances of being right.
• Use more than one model of potential across the business. The factors that are required for success are likely to vary at different levels of the business. For example, the characteristics that most accurately identify potential for advancement in executives will probably be different from those that predict potential in graduates. They may also vary in different functions or parts of the business.
• Answer the question, “Potential for what?” No one has potential that he or she will fulfill no matter what. We all need certain circumstances or environments in order to fulfill our potential. A polar bear may be an extraordinary animal, but it is not going to do well on a hot beach. Many readers probably know of people who have been successful in one job, only to fail in another. A common example of this is turnaround leaders. They typically have the drive and edge to pull a business back from the brink. Yet they may struggle once things have been turned around if they do not also possess the more nurturing qualities required to build and develop.

Wherever possible, then, make sure that measures do not just say how much potential someone has but also what he or she has potential for—that is, the types of roles and work environments individuals are most likely to succeed in. For managers who are simply rating potential as part of an annual appraisal, this may be an unnecessary complication. But for more formal methods and tools, it should be a requirement.

This latter point may surprise some readers. After all, it is commonplace now for businesses to rate people's general overall potential. Yet it is a critical point and touches on an issue that lurks behind all measurement: the importance of context. The fact is that whether someone succeeds is determined not just by how talented he or she is but also by things like the opportunities this person has and the business environment he or she is operating in. And as we will now go on to see, it is in this all-too-often-overlooked issue that we can find an important clue to how we can fundamentally change and improve our ability to predict success.

The Importance of Fit

The importance of the level of fit between people's talents and the demands of their jobs can be seen in a study that looked at the impact of General Electric (GE) leaders when they moved to new companies.⁹ GE is a particularly interesting example, since it deliberately tried to develop leaders with a range of experiences who would possess generic leadership skills that they could transport into any role. They were the personification of the “martini” manager, who would be good “anytime, anyplace, anywhere.” The market certainly seemed to believe this, anyway. In 85 percent of cases, the hiring company's stock price rose as soon as it was announced that a CEO from GE had been appointed.

The researchers, however, wanted to check if this faith was warranted, so they categorized both the strategic challenges facing each company and, using résumés, the skill sets of the former GE leaders (distinguishing between different types of leadership experiences). They then divided the CEOs into two groups. In one there was a good match between business need and the leaders' skill sets, and in the other there was a mismatch. They found that the performance of the businesses where there was a good match with the leader's skills was over double that of the mismatched group. So leadership skills do not appear to be as transportable as has sometimes been thought, and ensuring good person-job fit pays. Literally.

The vast majority of companies realize this. There may have been a time when picking the best people was all that mattered. These days, though, we generally try to pick the best people for a particular job. We have job descriptions to match people's skills against. Even when we are assessing people for developmental purposes, we usually evaluate them with an eye to how effective they will be in their roles. The problem, however, is that our judgments about fit at present suffer from two big flaws: they are too implicit and too narrow. Let's briefly look at each of these.

Four Useful Types of Fit

Matching people with roles usually involves four different types of fit (see figure 3.1):

• Person-job fit: The degree of fit between a person's qualities and the requirements of a particular role
• Person-organization fit: The degree of fit between a person's characteristics and the working environment or culture
• Person-team fit: The degree of fit between a person and the colleagues he or she will be working most closely with
• Person-manager fit: The degree of fit between a person and the manager she or he will be working for

Figure 3.1 Measuring Fit

A review of 172 studies found that these four types of fit were more or less distinct. It also found that they all matter.¹⁶ As we would expect, person-job fit is important for predicting performance, productivity, job satisfaction, and reduced job stress.¹⁷ Person-organization fit, by contrast, seems to be the best predictor of commitment, organizational citizenship behaviors, and staff turnover.¹⁸ Person-team fit has been less studied but appears to predict the quality of relationships with coworkers. And finally, person-manager fit predicts both employees' satisfaction levels with their manager and turnover.¹⁹ As the old adage says, people join companies but leave bosses.

It is worth pointing out that when we say “fit,” we do not necessarily mean “similarity.” For example, there is some suggestion that performance levels may be greater when managers and employees have different personality traits.²⁰ Likewise, diversity among team members may be a liability or an asset depending on the type of role.²¹ Where innovation or experimentation is important, diversity is likely to enhance performance. Yet for tasks focusing on production or execution, homogeneous teams may outperform diverse ones.²²

Three Steps for Measuring Fit

So what does this mean for how we approach talent measurement and for how we can turn talent data into talent intelligence? We have three key recommendations:

These suggestions may sound more daunting than they really are. Managers usually already make implicit decisions about each of these things.²⁴ What we are suggesting is making these judgments more explicit, visible, and informed. And we should be more explicit because being implicit is not working. A study of more than twenty-eight thousand newly hired employees found that businesses get the right person for the job on the majority of occasions but that they get the right person for the organization only 29 percent of the time.

This does not need to be complex. For example, as an easy minimum, organizations could require assessors and decision-making managers to provide a simple rating of each of these four types of fit. The fit ratings can then be added to give an overall score, or they can be weighted differently to emphasize particular types of fit.

Of course, it is possible to go further. Personality and cultural tests already exist that make it possible to compare individuals' personalities with the prevailing organizational culture or with a manager's own personality and ways of working. The globally used PAPI personality test, for example, comes with a job-profiler tool, which allows managers or job experts to complete a short questionnaire that determines the requirements of the role. The final test report is thus able not just to describe an individual's personality but also to show how this may or may not fit with the requirements of the role.

Indeed, the issue of fit seems to be where personality and character factors finally demonstrate the levels of importance in predicting success that we would expect them to. We showed in the previous chapter that personality tests have largely been disappointingly poor at predicting performance, at least compared to intelligence tests. But for measuring person-organization fit, person-team fit, and person-manager fit, personality is likely to be our best bet.

Moreover, there is some evidence that developing specific personality measures for a particular business may be more effective than using off-the-shelf generic personality measures.²⁵ Since this is not often currently done, there needs to be more research into the matter, and obviously it would make sense only for larger organizations. But as a longer-term option, it could be worth exploring.

Not every firm can or needs to go this far, though. The three core steps for using fit we have proposed are very simple, even simplistic. Yet they have the potential to fundamentally change how we think about measurement and significantly improve our ability to measure talent and thus also our people decisions.

The opportunity here lies in what we have traditionally used talent measurement for. When we are told that someone is a good performer, the most common response is usually, “Compared to whom?” or “At what?” And in general, the talent measurement market to date has been more focused on an­swering the first question than the second. It has largely concentrated, then, on trying to identify the brightest person, the most driven, and the best—hence, the current popularity of generic benchmarks that help businesses understand how good their people are compared to those in other firms.

Benchmarks can certainly be interesting and useful, but they have limits. Because by not more explicitly focusing on fit and sufficiently considering context, they leave out a large part of the picture. To develop real talent intelligence, achieve value-adding talent insight, and genuinely make some meaningful business impact with talent measurement, we need to be able to answer the, “At what?” question. The good news is that we can, and the key to doing so is the issue of fit.