Certifications as Benchmarks
Some industries use benchmarks, where if a person has some X, say a certification, then you can say that a person can achieve a certain level of performance. This is true of computer programmers who, once they possess certain certifications, are considered to possess certain skills, and thus they are competent to have certain jobs and to perform the tasks associated with those jobs. Recent years have seen a rise in demand for IT security certifications and an increase in the number of certifications being offered. This rise in demand has also been accompanied by a shift in the type of certifications available. Certifications were originally developed to sell professional services and were offered by vendors or resellers and thus driven by vendor interests. The job market is clearly the major driver behind certifications. Individuals seeking employment find it easier to get a job in the information security field if they hold a certification. Additionally, many employers offer pay bonuses for certificate holders. Certifications are basically Yes/No binary operators which function as metrics; either you have the certification or you do you do not. This is one of the ways that performance is measured in the IT industry.
However, it is possible that IT certifications do not measure performance and actually only measure how someone has done on a test. There are two potential routes we could take here. We could say (1) that other metrics of performance should be used in the IT industry, or (2) that the metrics used to generate and grade the tests are the metrics that would be really interesting to look at. Even if (1) is true it is possible that (2) is also true.
While there are several efforts under way to create a set of standards or best practices for IT security, no single consensus in this area has emerged. There is not, therefore, one agreed upon knowledge base around which certifications should be designed or one set of metrics to measure the competence of an IT security professional. With this lack of guidelines, each certification in the Information Security field has developed its own concept of what a security professional should know
Several specific requirements are worth mentioning as particularly relevant for IT security certifications. These include fair and equitable policies and procedures; documentation of how the certifying body is independent, impartial, and ethical in its decision making (“legitimacy” in Clark’s words); a structure that allows participation of all interested parties; if training is provided, evidence that it does not compromise the certification process and “teach to the test” (ANSI 1999); evidence of a formal process accepted by stakeholders regarding competence for entry into the field; and review of examination/assessment development process by a psychometrician.
There is another set of standards aimed solely at the “professional and technical issues of test development,” and test development could be thought of as little more than the development of metrics. Developed jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education, these are called the Standards for Educational and Psychological Testing. (APA 1999) They address how a test is constructed, evaluated, and documented, plus fairness in testing procedures and testing applications. These standards are incredibly detailed and technical, suggesting that the most practical course of action for many certification providers would be to hire a psychometrician or other professional familiar with such procedures to guide the test development process. A few of the highlighted themes in the Standards for Educational and Psychological testing include the importance of ensuring the validity of the test (that evidence and theory support the interpretations of test scores), the reliability of the test (that measurements are consistent when the test is given to many people), and the appropriateness of test statistics.
The existence of all these standards should not suggest that developing a sound methodology is simple or that there is little room for disagreement. Even in areas like the SAT (a form of measuring competency through examination, making it similar to many certifications), which has been administered to high-schoolers since 1926, questions are still raised. Despite the test’s longevity there is still debate over what we should look for in college applicants, whether the SAT measures any of those metrics at all, and whether it is biased. Many colleges and universities are even starting to make the SAT optional for applicants. All of this is to say that developing sound certification processes is difficult and not without debate, but this does not take away from the fact that best practices should be attempted and discussed. What this discussion of certification shows more generally for the discussion of metrics is that you have to make sure that you’re measuring performance and not the potential ability to do something. In addition, the discussion of how certification tests are generated gives us some insight into how metrics in general should be generated. They should be developed by external bodies using legitimate practices that the players buy in to.
Teachers. One thing that many people agree on is that the state of our nation’s schools is in trouble. Teacher evaluation and metrics is one of the most contentious areas where metrics are being used. It is incredibly difficult to measure the performance of teachers and it is also a political battleground, which makes the process even messier. However, generating these metrics can potentially aid the cause of educating our youth. Dan Condron, in testimony before the Subcommittee on Postsecondary Education, Training and Life-Long Learning, said, “We should provide incentive funding for states and districts to develop and implement value added or other systems for evaluating teachers based on student performance and who use their metrics to increase the skills of the existing teacher base.” (Condron 2000)
The question is how can we come to an agreement as to what the performance of a school should be and how it should be measured? The most high profile area in education where metrics have been used is in the No Child Left Behind Act (2001), which is part of what is known as the accountability movement. It “puts a priority on teacher quality, as determined primarily by the evaluation process. But in evaluating teachers for purposes of accountability, principals face the public’s perception that it is almost impossible to terminate a teacher for incompetent instruction. This perception is reinforced by professional literature which gives the impression that the courts place heavy demands on districts in cases involving teacher evaluation.” (Zirkel 2004) The No Child Left Behind Act (NCLBA) focuses on metrics used to measure the performance of schools. To this end “states must develop and implement a single, statewide accountability system that will be effective in ensuring that all districts and schools make adequate yearly progress, and hold accountable those that do not. States must specify annual objectives to measure progress of schools and districts to ensure that all groups of students—including low-income students, students from major racial and ethnic groups, students with disabilities, and students with limited English proficiency—reach proficiency within 12 years. States must set intermediate goals that provide for adequate yearly progress targets, with the first increase to occur no later than 2004–2005. In order to make adequate yearly progress, schools must test at least 95 percent of their students in each of the above groups.” (U.S. Government 2001) The accountability movement might have wanted only to focus its aim at the school level, but once schools were supposed to be held accountable then performance metrics for teachers were inevitable. Another result of the NCLBA is that a market has sprung up for performance metrics services for schools.
One company that provides a solution to the teacher performance metrics is SPSS. SPSS Performance Metrics Jumpstart claims to be able to help “you learn how to develop a measurement system that delivers accurate performance data on your programs, services, and courses.” (SPSS 2006) It does this by providing users the skills to: “Justify and maximize your budget appropriation using reliable performance data, measure participant satisfaction with programs and services via a survey design system, monitor program effectiveness by establishing key performance indicators (KPIs) and key performance predictors (KPPs), maximize program and service delivery by deploying KPI and KPP models throughout your organization, isolate predictors of superior service delivery through analysis of data stores, and react quickly to performance problems via an early warning system.” (SPSS 2006) Such products and companies provide a great service in terms of teacher performance metrics.
However, the main problem with teacher performance metrics is that people (experts included) can’t seem to agree on what these metrics should measure. Therefore, the products that are being marketed now might be used across a school district (which is a fairly high level of standardization) or they might just be used in one school (which is a fairly low level of standardization). As a result of the lack of standardization in teacher performance metrics a teacher that scores well on one performance metric might not perform well on another. This leads to confusion (and perhaps arises out of a more basic confusion, which is that we don’t agree on what the performance of a good teacher is supposed to look like).
What this means for the generation of HR performance metrics is that it is in our interest to come up with industry-wide standards for HR performance metrics, for only then we will truly be able to reap the benefits of such metrics. In order to generate such standardized metrics, we have to agree on what performance in human resources means. This task requires a great deal of introspection and discussion.
One nonprofit organization that helps standardize a variety of metrics is the American National Standards Institute (ANSI) which “coordinates the development and use of voluntary consensus standards in the United States and represents the needs and views of U.S. stakeholders in standardization forums around the globe.” (ANSI 2006) Use of such an organization or the development of a new organization that helped industries standardize their metrics would be extremely helpful for the development of performance metrics.
Further, people can’t agree on what to do with performance metrics once they are generated. The main question here is whether they should be used to determine teacher’s pay. At issue is what is known in the education world as Performance Pay. The way teacher pay primarily works is that teachers climb step by lockstep up a traditional pay ladder, which entails automatically earning salary increases based on their education level and their years of service without reference to their performance on the job. However, “the trend toward performance-based pay for educators is a growing one.” (Rotherham 2000) Already “30 out of 50 states passed legislation requiring some type of performance pay for teachers, or some portion of teacher pay,” Dr. Marc J. Wallace Jr., founding partner of the Center for Workforce Effectiveness, is quoted as saying. (Delisio 2003)
People in favor of performance pay claim that implementing such schemes “will attract more people to the teaching profession and make those in the profession work harder.” (Delisio 2003) On the other side of the fence are those who argue that “any type of system tying salaries to teacher performance or student outcomes is flawed because of a lack of objective observers and objective criteria to evaluate a teacher’s performance.” (Delisio 2003) However, those on this latter side of the fence are perhaps merely saying that adequate performance metrics have not been generated. But such standards could possibly be generated if people came together to create metrics that were “credible,” “relevant,” and “legitimate.” And indeed, this is one of the mistakes that people make about metrics, that they must be objective to be worthwhile. In fact, very few people will ever believe any performance metric is objective. What we need are metrics that are “credible,” “relevant,” and “legitimate.” Metrics meeting those criteria can only be generated if experts and practitioners in the industry at hand can come together to discuss what performance in that industry is supposed to mean. This lesson is no less true for human resources than it is for education.
In addition, those who are opposed to performance pay note the possibility that such measures of output could be manipulated. (Rothstein 2000) This points to yet another lesson for those generating performance metrics. These data outputs in these performance metrics should be subject to high data protection standards, not merely so that those metrics will not be manipulated (although this should be reason enough) but also so that the firms generating such metrics are not held civilly or criminally liable for security breaches that allow the data collected to be compromised. Such data protection regimes should be instituted before performance metrics are put in place.
Crime. Metrics are, of course, necessary not only for the good, but also for darker elements as well. Crime is one area where metrics are needed, but also where they are strangely difficult to come by. How do we measure the performance of law enforcement in various communities? One of the ways we do this is via crime statistics. Crime statistics are compiled by two major sources for overall national crime statistics, the Uniform Crime Reports (UCR) and the National Crime Victimization Survey (NCVS).
The UCR began in 1929 following an initiative of the International Association of Chiefs of Police (IACP). The UCR is a measure only of crime that is reported to the police. As reporting in the UCR Program is voluntarily done by law enforcement agencies, the UCR lacks standardization across jurisdictions. Standardization is one problem that runs through any study of metrics. One fix to this problem is to have an external body manage the metrics for an entire industry.
The problem with the lack of standardization in crime statistics means that certain crimes are classified in different ways in different communities, therefore altering the crime statistics for that community and how the public views the performance of law enforcement officers. It should be noted that changing these crime statistics can have immense political consequences (and can be equally instigated by political motivations). For instance, charges of manipulation of crime statistics have been brought in New York City, Atlanta, Boca Raton, and Philadelphia, resulting in several resignations and dismissals as evidence of political tampering of numbers was uncovered in these cities. (Butterfield 1998) These cases are just a reminder that performance metrics can come under immense political pressure. Such pressure to change the data can come from governmental agencies, from the management within a firm, or from outside the company. If the pressure is given in to, the performance metrics used become meaningless.
The NCVS is a national survey of 42,000 households regarding incidents of victimization. Since it is a survey of the population, the NCVS provides an account of crime whether or not it has been reported to the police. So while it should provide a count similar to the UCR, it is somewhat different due to the fact that the UCR is reported by police and the NCVS is reported by citizens. The difficulty with the NCVS is that some respondents may not remember particular incidents or specific details of crimes, or they may conflate two crimes, or they may “remember” crimes that have happened to other people. The lesson for performance metrics here is that we should not forget that people’s psychology plays a role in how they answer questions, which might be a part of performance metrics. This is perhaps particularly true in human resources, where individuals might be answering questions about colleagues about whom they have complex feelings.