Table 8.1 Competency Requirements of the Five Core Measurement Roles

Table 8.2 Examples of the Two Groups of Vendors

Conduct a Proper Selection Process.

One of the most common mistakes companies make when selecting vendors is not following a proper process. Lacking technical understanding and faced with an overwhelming number of options, many HR and business leaders have a favorite vendor or tool that they seek to use whenever possible. The selection process can thus become a cover for selecting a preordained favorite. This is bad business in any procurement process, and no less so for measurement. Without genuine competition among vendors, the likelihood of achieving high-quality, cost-effective solutions is distinctly reduced. So follow a proper process. The precontest favorite may well still win, but the competition will put the true value of its bid in perspective.

Select a Measure, Not a Vendor.

A related issue is that because they lack expertise in the field, some businesses simply choose the vendor with which they feel they have the best chemistry. From a business's point of view, it is not entirely a bad idea. One of our core rules is to ask ourselves whether, if things go horribly wrong, we would trust a vendor to stand beside us and deal with the problem. This kind of trust or chemistry factor is a valid selection criterion, but it must not be the only or main one. Organizations must also investigate the quality of the products or services they are purchasing. This is partly about making sure that what they buy measures precisely what it is intended to. Chapters 2 to 4 should help in determining this. It is also partly about questioning the validity of measures. This is especially important given the increasing evidence of reporting bias: the tendency for some vendors, while presenting validities as scientific, to report these figures selectively in a way that overstates the effectiveness of their tools. The guide to questioning validity we presented in chapter 5 can help here. No matter how you do it, though, it is important not to assume that measures are effective or to take vendors' claims at face value. Always question them.

Look for Expertise.

Although we have just said you should focus more on the measure than on the vendor, one critical aspect of vendors is their level of expertise. With individual psychological assessment in particular, it is arguably more im­­portant to choose the right assessors than to focus on the measurement process. (For more on how to select an individual psychological assessment vendor, see the appendix.) With smaller vendors, look for people who have come from bigger, more established providers. For example, the founders of the UK-based VesseyHopperMcVeigh all learned their trade at SHL. In larger vendors, individuals' backgrounds are also important, but another useful source of insight can be the origins of the business overall, since mergers and acquisitions can hide who you are really buying from. For instance, IBM's acquisition Kenexa is more known for its talent acquisition systems, but in 2006 Kenexa purchased a reputable psychometrics firm called PSL. And although some people may not have heard of Cubiks (one of the larger global vendors), its origins lie in PA Consulting Group, one of the world's best-known management consultancies.

Reputations and backgrounds are not always what they appear to be, of course. One or two large psychometrics vendors seem to have dined out on their reputation for quite a time now, even though their tests have since been overtaken by newer, better tools. Yet for businesses that do not have technical expertise and need to acquire it, reputations can be a useful indicator.

Do Not Select Only on the Basis of Price.

At the risk of repeating ourselves, measurement is a precision instrument. Purchasing only on price is like buying a supercar on the cheap and still expecting it to be super. There is a degree to which the cost simply is what the cost is. For example, we have seen many organizations seek to drive down the cost of individual psychological assessments, only for vendors to respond by putting less experienced, cheaper consultants in place. You get what you pay for. This does not mean that price is not important. Businesses obviously need to secure the most cost-effective solution possible. But those that select only on the basis of price invariably are stung on quality.

Intellectual Property Rights.

A number of years ago, one of the best-known global measurement vendors signed a contract with one of the world's largest oil and gas suppliers to produce a new leadership framework for them. Unfortunately the energy firm either did not check the details of the contract or did not understand what they meant: the contract assigned the intellectual property rights (IPR) in the framework to the vendor and prevented the business from using the framework with any other vendors. As a result, the firm was effectively tied to using that vendor for all measurement activity at leadership levels. The only way it can stop using the vendor is to create a new leadership framework.

Intellectual property rights are vital, and some of the biggest vendors are some of the worst culprits here. There are two simple guiding rules. First, if a business pays for the design of a new measure that is not based on something in which the vendor already owns the IPR, then the IPR for the new measure should belong to the business. No debate, no exceptions. If a business pays for the development of a new 360-degree feedback tool or situational judgment tests, then the questions that the tool comprises should belong to the business. The IT system that the tool is delivered on still belongs to the vendor, but the questions should belong to the business. By all means, share the IPR for the questions with the vendor, but do not merely give them to it. There should be no ties that bind. If a vendor does not do its job well, firms should be free to walk away. Moreover, if a vendor develops a tool for you at your expense, do you really want it using some of that content with one of your competitors?

Second, businesses should retain IPR over all data and results but share this unconditionally with the vendor. So if a firm pays to use a psychometric test, it should have the IPR over all the results and reports produced. However, since vendors use these data to produce norm groups and studies, organizations need to share the intellectual property with vendors. This is often done by granting a “perpetual, royalty-free license to use data in an anonymous form for the purpose of norm generation and research.” The principle is that both parties should be free to use the data for their respective purposes. Sometimes vendors do not want to provide data to a business, for example, when an assessment has been part of a developmental process. But in our view, as long as the data are kept securely within the business's central measurement team and confidentiality is maintained, if the business has paid for the data, it should be able to use it.

Intellectual property, then, is an important issue. At stake is a company's freedom to choose to work with another vendor or walk away if a vendor does not do a good job. This freedom is so fundamental that we have one overarching piece of advice: if a vendor insists on retaining IPR over something new that you pay it to design or if it is unwilling to share IPR on data with you, find another vendor that will. No debate, no exceptions.

Total Price.

In principle, this ought to be easier than the intellectual property issue, but it is not always. Businesses need to be clear not only on the upfront costs of using a measure but also on the total costs across the lifetime of a contract. Obviously, at the beginning, it is often not certain how much a measure will be used. But the extra add-on costs that can emerge need to be specified in advance. For example, project management fees increasingly seem to be viewed as an add-on. We experienced one vendor showing these costs in a brief comment in small print at the bottom of its proposal: “A 14 percent project management fee will be added to all costs.” It claimed that it had separated out the cost to make the charges easier to understand, but we were not convinced by this, and indeed remain unconvinced.

Other common add-on costs are things like training end users to use reports (which some vendors may insist on) and training staff to use test administration systems (which may be necessary). With psychometrics, be aware of any test development that may be required, such as additional translations or the development of norm groups specific to your business. Finally, some vendors charge extra for attending annual or quarterly review meetings, although we recommend that you do not accept any such charges.

The key here is to be aware of potential additional costs and that some vendors are more forthcoming and clearer in presenting them than others.

Travel Claims.

This is not usually a contentious issue, but it is one to be aware of. The idea here is that vendors should not charge for the time it takes them to travel to and from any location to complete work for a business if that travel lasts less than a certain number of hours (door-to-door). There are regional variations in what this number is, but it is usually between two and four hours. When travel takes longer than this, the total time spent traveling is typically charged at 50 percent of consultants' normal day rate.

Targets, Performance Indicators, and Penalty Clauses.

All contracts should lay out clear targets, performance indicators, and penalties. Organizations that do this well tend to report higher levels of vendor performance and lower levels of opportunistic behavior from vendors. The bit that is often missed is the penalty clauses. Yet they are essential, since they set the boundaries of acceptable performance. Bear in mind, though, that this is a two-way street. Vendors may well want to place penalty clauses of their own, and for the most part, we support this. Contracts, after all, need to be fair to all parties.

Termination Clauses.

Last, but by no means least, there should be no penalty if a business terminates a contract. Sometimes a vendor offers a reduced setup or design fee to attract businesses, but it compensates for this by applying higher fees per assessment or by assuming that it will receive a certain amount of business. In these situations, it might want a termination clause that ties businesses in for a certain period and thus gives the vendor a chance to recoup its setup costs. In our experience, it is better to pay a higher setup fee and be free to walk away if thing go wrong than to choose lower setup costs at the expense of being tied to a vendor.

Onboard Vendors.

Without an onboarding plan, it can be difficult to ensure that vendors have the necessary level of understanding about the business and its culture to develop and deliver effective solutions. With methods such as assessment centers and individual psychological assessment, for example, assessors need to understand the business if they are to make accurate judgments about individuals' level of fit with it.

If you are using these methods, it is important to arrange an induction to the business for the vendor's team of assessors. The introduction should familiarize them with the company's strategy and culture and advise on the types of people who tend to do well in the business: that is, those who tend to fit in well with the culture, perform well, and be promoted. It is also important to arrange a conversation between the assessor and the manager or hiring manager of an individual being assessed. The manager can then advise on the challenges the individual will face in the role and the stakeholders he or she will have to work and interact with.

Obviously, with measures such as psychometrics, these steps are not so relevant. Yet for processes in which an assessor is making judgments, they can be critical. So we are continually surprised by the lack of attention paid to them. Many businesses seem to assume that vendors have business knowledge and therefore skip this step. Yet by doing so, they risk limiting the effectiveness of any measures, since without these briefings, assessors cannot accurately measure fit.

Evaluate Vendors.

In early 2011, the Romanian parliament sat down to discuss a controversial new law: that Romanian witches and fortune-tellers making predictions about the future should be penalized if their predictions did not come true. At the risk of being cursed, we think this is a good idea, and not just for witches. Measurement vendors are effectively making predictions about the future—about who is likely to succeed and who is not. Yet businesses all too often purchase measures without subsequently checking if the measures genuinely are predictive. Indeed, we never cease to be amazed by the number of firms that speak highly of their vendors but have no hard, internally produced data to back this up. Since firms invariably go on to purchase more measures, this is extraordinarily bad business.

This is why in chapter 6 we described the need to evaluate the impact of measures as one of the four foundations of making measurement work. And as we noted there, this is more than about simply checking whether measures work after the fact. Measuring effectiveness is a key element of tailoring tools to businesses' specific needs, and if this not done, organizations are limiting the effectiveness of measures.

Leverage Market Forces in Managing Suppliers.

Larger organizations often use multiple vendors, yet few have an active plan to leverage the market forces that can be created by having more than one vendor. This is a lost opportunity, since research shows that the market forces created by competition between vendors can result in improved service levels and reduced costs.¹ There are three things in particular that businesses can do here: do not let the vendor pool grow too large, measure vendors' performance, and make these performance data widely available.

For example, when setting up an individual psychological assessment process for new senior-level staff across a large multinational energy firm, we realized that the sheer number of locations meant that we were going to need more than one vendor. We therefore launched what we called a guided calibrated market. We selected three vendors and agreed on a common format for the IPAs, a common process for scheduling them, and common, standardized reports. We also used standardized contracts to ensure that the terms and conditions were the same for each vendor. The only element that differed between them was the price. We then gave the various business units the choice of which of the three vendors to select. We brought the vendors together regularly to calibrate judgments and tracked their rating tendencies statistically. We also measured their performance along a number of dimensions and made this information available to the business units on an ongoing basis. Finally, at the end of every year, we evaluated whether to retain the vendors or replace some or all of them. The business loved having the choice, and we are certain that we achieved a better-quality process and lower costs by adopting this approach.

Beware Closed Systems.

One way organizations seek to control the quality of vendors across diverse business units is to use preferred supplier lists. Although these lists have their benefits, it is critical to have a mechanism for promoting new suppliers onto the list and removing existing suppliers from it. Without such a mechanism, businesses' ability to apply market pressures to vendors will be limited.