3

On Rigid Decision Rules for scholarships

The Preliminary SAT / National Merit Scholarship Qualifying Test (PSAT/NMSQT) is a program cosponsored by the College Board and the National Merit Scholarship Corporation. It's a standardized test that gives firsthand practice for the SAT. It also provides an opportunity to enter National Merit scholarship programs and gain access to college and career planning tools. It is taken by more than 1.5 million students annually.

The PSAT/NMSQT is designed to measure critical reading skills, math problem-solving skills, and writing skills. It is made up of relatively easy, retired SAT items and is dirt cheap to construct and administer. It also figures prominently in the third conclusion of the National Association for College Admission Counseling's report in September 2008 on admissions testing. The NACAC recommends that the use of the PSAT with a rigid qualification cut-score for such programs as the National Merit scholarships be immediately halted.

The NACAC supports this recommendation (demand?) by pointing out that requiring a cut-score to qualify for a scholarship (and by generalization, for anything) leaves no room for human judgment at the boundary. All sensible people would agree that there is insufficient information to conclude that someone just above the decision boundary is more worthy than someone else just below it.

Yet with limited resources, triage decisions must be made.

The problem with a hard-and-fast cut-score is one that has plagued testing for more than a century. The Indian civil service system, on which the American civil service system was based, found a clever way around it. The passing mark to qualify for consideration for a civil service position was 20. But if you received a 19 you were given one “honor point” and qualified. If you scored 18 you were given two honor points, and again qualified. If you scored 17, you were given three honor points, and you still qualified for further consideration. But if you scored 16 you did not qualify, for you were four points away. I don't know exactly what the logic was behind this system, but I might guess that experience had shown that anyone scoring below 17 was sufficiently unlikely to be successful in obtaining a position, that it was foolish to include them in the competition. But having a sharp break at 16 might have been thought too abrupt, and so the method of honor points was concocted.

How does this compare with the Merit Scholarship program? The initial screening selects 15,000 (the top 1 percent) from the original pool of one and a half million who take the PSAT. These 15,000 are then screened much more carefully using both the SAT and ancillary information to select the 1,500 winners of a scholarship (the top 10 percent of the 15,000 semifinalists). Once this process is viewed as a whole, several things become obvious:

1. Since the winners are in the top 0.1 percent of the population, it is dead certain they are all enormously talented individuals.

2. Many worthy individuals are missed, but that is inevitable if there is only money for 1,500 winners.

3. Expanding the initial semifinal pool by even a few points will expand the pool of semifinalists enormously (the normal curve grows exponentially at that point), and those given the equivalent of some PSAT “honor points” are extraordinarily unlikely to win anyway, given the strength of the competition. It would also increase the cost of the final screening apace.

What about making the screening more rigorous—rather than using the PSAT scores alone? Such a screening must be more expensive, and to employ it as widely would, I suspect, use up much more of the available resources, leaving less money for the actual scholarships. The irony is that utilizing a system like that proposed by the NACAC would either have to be much more limited in its initial reach, or it would have to content itself with giving out many fewer scholarships.

Of course, one could argue that more money should be raised to do a better job in initial screening. I would argue that if more money were available, the current method of allocating it should be continued and used to either give out more scholarships, or bigger ones.

This completes the explication behind my initial conclusion that some of the recommendations of the NACAC make sense only if you say them fast. Relying on evidence in the evaluation of policy proposals makes decisions wiser, if for no other reason than it forces you to slow down a bit.

In the past three chapters I have presented evidence that a general aptitude test has many advantages over suggested alternatives. As an illustrative example I have used the SAT and its junior partner, the PSAT. In chapter 4 I will go further and show in some detail how the use of such tests can provide important help in the triage decisions that are inevitable in public education, when limited resources must be allocated to overwhelming need.