Chapter Four

College Rankings Go Global

It would be hard to overstate just how contentious rankings of U.S. colleges and universities have become in recent years. As editor of the U.S. News & World Report college guides in the mid-2000s, I became accustomed to a steady stream of college presidents and admissions officials visiting the magazine’s Washington, DC offices to complain about the publication’s influential college rankings. Some thought outsiders—especially journalists—shouldn’t be ranking colleges at all. Some took exception to one or more aspects of the methodology used to calculate each college’s standing. Others insisted that if only their own data had been properly tabulated by the magazine, their ranking would be higher. Outside the magazine’s offices, of course, those that fared well in the rankings often trumpeted the results on their Web sites.

Several years later, the rankings debate had spread far beyond United States. In the fall of 2007, I listened intently—from the sidelines, mercifully—as the elegantly dressed director of the École Normale Supérieure, Monique Canto-Sperber, addressed an international conference hosted by Shanghai Jiao Tong University. An expert in Greek philosophy, she delivered a meticulous demolition of the university’s closely watched global college rankings, explaining why they couldn’t possibly do justice to the strengths of her celebrated institution. The only difference from the debate in the United States was that U.S. universities at the very top of the pecking order tend to publicly ignore the rankings, whereas in Shanghai the president of an elite institution made clear that, for her university, the global rankings are a serious business indeed.

The colloquy that took place in Shanghai is by no means an isolated one. In recent years, college rankings have experienced explosive growth around the globe, as the same calls for accountability and objective evaluation seen in American higher education have become ubiquitous. Rankings now exist in more than forty nations,¹ whether established by journalists, government agencies, nonprofit organizations, or universities themselves. There are global rankings too, of course. All are closely watched and are also a source of great controversy, in part because of the serious consequences that sometimes result for universities that are deemed not to measure up. All this should come as no surprise. Students everywhere increasingly behave like consumers. At the same time, governments closely follow the extent to which the universities they fund are contributing to innovation and national economic growth. As students become more mobile, universities move across borders, and competition for world-class status becomes ever more intense, little wonder, then, that college rankings have gone global. They are an unmistakable reflection of global academic competition. Many would say they fan that competition as well. Rankings seem destined to be a fixture on the global education scene for years to come. Detractors notwithstanding, as they are refined and improved they can and should play an important role in helping universities get better.

BIRTH OF THE RANKINGS

In the United States, whose college rankings have been uniquely influential, a number of efforts at evaluating colleges long predate those inaugurated by U.S. News in 1983. Some, intended earnestly, look rather whimsical in retrospect. In the 1895 Illustrated History of the University of California, a detailed chart² offers a “before” and “after” snapshot of the fitness level of the men of the then-young university, comparing the physical prowess of its undergraduates to those at three of the university’s well-established East Coast rivals: Yale, Amherst, and Cornell. Upon entering the university, the illustrated chart demonstrates, the average UC man had biceps, chest, thigh, calf, and arm measurements, plus strength in various body parts, well below the average of a sample of 15,000 of his blue-blood competitors. But following two years of calisthenics in the healthy California air, the chart shows, brawny Berkeley students had surpassed their effete East Coast counterparts. Perhaps this early cross-college comparison is more analogous to today’s intercollegiate sports ratings than to any academic ranking. Still, its focus on how much students improve while in college is striking for the way it foreshadows the value-added analysis that is much sought after in today’s higher education world.

That outcomes-oriented approach was only hinted at in the earliest truly academic college rankings in the United States, which date back to the turn of the twentieth century.³ They focused simply on which universities produced the most distinguished graduates, following the example of an Englishman named Alick Maclean. In 1900 Maclean published a study called Where We Get Our Best Men, which looked at the characteristics of the eminent men of the day, including nationality, family, birthplace, and university attended. In the back of the book, he published a list of universities ranked by the number of their prominent alumni. The first U.S. ranking in 1910 took a similar “reverse-engineering” approach, examining successful individuals and crediting their alma maters for their eventual success. A number of different rankings were attempted over subsequent decades, most looking at graduate-level education and almost all continuing the emphasis on either some version of the “great man” theory of educational quality, or, in one case, of how well a college’s graduates performed in graduate school.

By the early 1960s, however, survey-based reputational methodology began to supplant the rankings’ earlier focus on outcomes. Soon, almost every published ranking was based on an institution’s reputation among its peers rather than the accomplishments of its graduates. Interest in examining undergraduate education reemerged during that time, too. And with the emergence of commercially successful rankings such as Allan Cartter’s discipline-by-discipline Assessment of Quality in Graduate Education, which sold 26,000 copies and received great critical acclaim, reputation-based college rankings began to come into their own.

Nevertheless, rankings were still scrutinized mostly within the guild of academe rather than in the wider world of college-bound students and their families. That would soon change. By the late twentieth century, attending college had become a mass phenomenon. By 1972, nearly half of all high school graduates continued directly to either a two- or a four-year institution.⁴ And while most U.S. undergraduates have always attended public institutions near their homes, the growth of meritocratic admissions at the nation’s elite universities led to growing competition for admissions slots. Where selective colleges were once fed largely by a small number of Eastern prep schools, the opening of those colleges to a broader swath of society—beginning in the 1960s and continuing at a faster pace during the 1970s—led to wider interest in how to make a sound college choice, and, in an increasingly competitive admissions climate, how to get in.

Against this backdrop, a number of different efforts emerged to inform students and parents about their college choices. One, The Gourman Report, was published from 1967 to 1997, ranking colleges to within two decimal places despite having a methodology that was shrouded in mystery.⁵ A far more credible effort was launched in 1981 by Edward B. Fiske, then education editor of the New York Times, under the title The New York Times Selective Guide to Colleges (a name that was later changed, when the guidebook became too controversial for the Times, to The Selective Guide to Colleges and then The Fiske Guide to Colleges). Like its main competitor, the Yale Daily News’s Insider’s Guide to the Colleges, Fiske’s guide was compiled with the help of a network of student reporters at campuses around the nation. It profiled several hundred colleges, using a narrative feature-article style to paint a picture of academics and student life at each university. The guide, which remains popular today, uses a version of the Michelin star system to rate the social life, academic rigor, and “quality of life” at each campus, with a maximum of five points available in each category. Perhaps because it is a rating rather than a ranking—there is no ordinal numbering of top, middle, and bottom colleges, and many colleges can earn a four- or five-star rating in a given category—it has never been as controversial as other efforts to see how colleges stack up against one another. “I originally got a lot of flack, since colleges were used to doing the judging of students, not being judged,” Fiske says. “But U.S. News, bless its heart, took the pressure off me. People love rankings, which is not really what I do, and the way people use the rankings really irks colleges and, for that matter, college counselors.”⁶

Indeed, the U.S. News rankings were contentious from the very start. Launched in 1983, the rankings began simply enough as an outgrowth of another journalistic project for which the newsmagazine had received significant attention: a survey asking U.S. leaders to identify the most influential Americans. In a detailed account written by the late Alvin Sanoff, the longtime managing editor of the rankings project,⁷ he says the magazine’s editors at the time were hoping to gain similar notice—and, to be sure, sell some magazines—with an effort to identify the best colleges in the United States. Although today’s U.S. News rankings are often criticized for focusing too heavily on reputation rather than actual accomplishments, the magazine’s first rankings were exclusively based on reputation. U.S. News surveyed college presidents around the country, asking them to identify the nation’s best universities. It published the results in a regular issue of the magazine, first in 1983 and again in 1985. It was only in 1987 that the magazine published a separate guidebook entitled America’s Best Colleges, extending its reach to include not only undergraduate education but also law, business, medical, and engineering schools.

The debut of the rankings was well timed, coming at the height of the U.S. consumer movement. “A generation of parents who were college-educated brought both pragmatism and status-seeking to the college search process,” writes Sanoff. “While many members of earlier generations were simply pleased that their children were going to college, members of the Baby Boom generation cast a more critical eye toward higher education. They wanted value for their money.” The first generation of U.S. News rankings catered to that appetite in a fairly simple way. The presidents surveyed were asked to pick the ten colleges that provided the best undergraduate education in the academic category to which their university belonged. These nine categories, based loosely on the so-called Carnegie Classifications established by the Carnegie Foundation for the Advancement of Teaching, included National Universities, National Liberal Arts Colleges, Southern Comprehensive Institutions, Eastern Liberal Arts Colleges, and so forth. In the first two, most closely watched categories, the magazine published a ranking of the top twenty-five institutions. In the remaining categories, it ranked the top ten. This straightforward early effort—a fairly standard journalistic approach to a consumer-advice story—had within a few years become more complex, more popular, and more contentious. “No one imagined that the rankings would become what some consider the 800-pound gorilla of American higher education,” writes Sanoff, “important enough to be the subject of doctoral dissertations, academic papers and conferences, and endless debate.”

When the first full-fledged U.S. News guidebook was published in 1987, a delegation of college presidents and senior administrators met with the magazine’s editors and asked that the rankings enterprise be stopped. Purely numerical indicators are an inappropriate way to measure the varied institutions and missions of U.S. higher education, they argued (as would umpteen critics in years to come). Moreover, the reputational survey of college presidents amounted to nothing more than a beauty contest. The U.S. News editors listened, but rather than do away with their already flourishing enterprise, they made significant changes to the rankings instead. As Sanoff recounts, they consulted with outside experts and decided to divide the rankings into two components. The reputational survey was revamped to include not only college presidents but also provosts and deans of admission who would bring a broader base of expertise to the task. At the same time, a range of quantitative measures was introduced, each requiring data collection followed by the application of a weighting determined by the magazine’s editors. The objective data included several components: (1) student selectivity measures, such as a college’s acceptance rate and the average SAT or ACT scores of the entering class; (2) student retention data, notably the graduation rate and the percentage of first-year students who returned for their second year; (3) institutional resources, mostly consisting of research spending; and (4) faculty quality, including average faculty salaries and the percentage of faculty with PhDs.

In one form or another, the basic structure of the rankings has remained roughly similar ever since, notwithstanding the addition of a few supplementary factors, changes in data sources (the quality of the magazine’s data is widely regarded to have improved), and tweaks to the weighting of each category. U.S. News has also made a concerted effort to be more transparent with its methodology and to take suggestions from critics. The magazine invites a group of admissions deans to its offices every year to serve as an informal advisory board. And despite bouts of defensiveness, it does make changes that it believes improve the rankings. One example came in 1997. Detractors had argued for years that the rankings focused excessively on “inputs” rather than “outputs” by giving colleges credit for admitting well-qualified students rather than for how well they actually educated those students. In response, the magazine expanded a so-called value-added measure (now referred to as “graduation rate performance”), accounting for 5 percent of the total ranking in the “National University” and “National Liberal Arts College” categories, which assesses graduation rates on a curve, taking into account students’ incoming qualifications and socioeconomic backgrounds. This measure of persistence controlling for social and academic backgrounds doesn’t really capture how much students learn—an elusive question that is in many ways the Holy Grail of college rankings—but it does reflect an effort to fine-tune the rankings where possible.

Still, criticisms of the rankings have continued unabated. Colleges eager to present their numbers in the most flattering possible light have frequently been accused—sometimes with good reason—of gamesmanship or outright fraud. In one recent instance, the former director of Clemson University’s Office of Institutional Research declared at a June 2009 conference that senior administrators at the school had deliberately given low “reputational” scores to rival universities in their zeal to help Clemson rise in the rankings. (The officials denied the charge.) Fierce battles have also been waged over methodology. Detractors note, among other things, that because of the inclusion of research spending and average class size (which is expensive for colleges to achieve), the rankings invariably reward well-endowed private institutions and punish public universities—many of which, ironically, were ranked far higher when U.S. News considered only reputation rather than including quantitative variables. More generally, the argument goes, the factors used in the rankings provide colleges with perverse incentives to focus on the factors that are measured rather than taking on the more elusive task of providing the best possible education for their students. “The U.S. News methods are really indefensible,” says Fiske. “They ask the wrong question. The issue is what is the best college for any particular student, not what is the best college in some abstract sense. . . . You cannot quantify the really important questions like matters of fit.”

In a sense, the U.S. News editors who designed the rankings became paradoxical victims of their own earnest efforts to make the rankings more rigorous. What began as a fairly conventional journalistic parlor game—a simple survey seeking the views of college presidents—had morphed into an exercise that took on many of the trappings of social science, with nationwide data collection, a detailed methodology, regression analysis, and so forth. But at heart, the rankings remained a journalistic rather than an academic exercise. Thus, it was not much of a surprise when the Chicago-based National Opinion Research Center, an outside consulting firm hired by U.S. News in 1997 to evaluate the ranking methodology, reached a conclusion that gave further fuel to critics. “The principal weakness of the current approach is that the weight used to combine the various measures into an overall rating lacks any defensible empirical or theoretical basis,” it said. To be sure, the report also noted that the weights might not be off-base. But it said they could not be defended “on any grounds other than the U.S. News staff’s best judgment on how to combine the measures.”⁸

U.S. News editors have continued to defend the rankings vigorously, challenging the magazine’s role as stock villain in the competitive world of college admissions. They argue that the characteristics measured—from research spending to class size to graduation rates to alumni satisfaction to qualifications of incoming students—are all factors that prospective college students and their families might reasonably want to know about. And they point out that the rankings always come with a cautionary note and accompanying articles that stress the importance of finding the right “fit” in a campus, and tell students they should use the rankings only as a starting point. Whatever ranking decisions the magazine makes, it is left in a no-win situation: When it changes the methodology, it is accused of seeking to sell more magazines by shaking up the previous year’s pecking order. When the formula goes unchanged from the previous year, the magazine remains vulnerable to the charge that it has done nothing to correct the rankings’ many imperfections.

Moreover, while the rankings certainly have many flaws, some popular critiques have proved to be urban legends. For years college officials alleged that U.S. News, by including “yield” as a factor in its selectivity measure, was stoking the surge in binding “early decision” admissions programs. Early decision requires students to attend a college to which they have applied and been admitted early. Yield refers to the percentage of students offered admissions who decide to attend. Early decision is controversial because it has often been blamed for adding stress to an already frenzied admissions process and for disadvantaging low-income students, who lose the ability to compare financial-aid packages from multiple schools in the spring when they are required to say yes to an early admissions offer in December. Because binding early decision by definition results in 100 percent yield, the use of yield by U.S. News was said to give colleges an incentive to boost their yield stats by relying more heavily than ever on early decision. This argument had one flaw, however: first, yield counted for only 1.5 percent in the ranking methodology. More important, when the magazine decided to remove itself from the controversy by removing yield from the ranking equation, early decision continued unabated. Rarely did critics acknowledge that universities have many self-interested reasons for maintaining a practice that, in an age when “enrollment management” is extremely important, permits them to predict more accurately the size and composition of incoming freshman classes.

As the debate continues, one thing is certain: more than twenty-five years after they were inaugurated, the rankings are routinely cited as having transformed the world of U.S. college admissions. In 2009, for instance, the University of Chicago announced the retirement of Theodore “Ted” O’Neill, the university’s longtime dean of admissions, who is best known for introducing Chicago’s quirky “uncommon application,” which asks applicants to write essays on offbeat topics. (A sample question: “Chicago professor W.J.T. Mitchell entitled his 2005 book What Do Pictures Want? Describe a picture and explore what it wants.”) O’Neill presided over a huge increase in applications as Chicago, notorious for its demanding academics, worked hard to expand its recruiting net to a broader range of students. But he bemoaned the increased pressure on universities—citing U.S. News in the process. “At some point, we were put on this nationally observed competition for numbers,” O’Neill said. “U.S. News did that, or we did it to ourselves. There’s this grander competition that is about numbers and ratings. Some of us resist that, some go along with it, but it affects us all.”⁹

The U.S. News college rankings were preceded by, or in some cases joined by, a vast array of other college guides and rankings, including the silly (the Princeton Review’s annual “top party schools”); the cut-and-dried (the phonebook-sized Barron’s, Peterson’s, and Lovejoy’s guides); the niche-oriented (guides identifying the best colleges for African Americans, Christians, and conservatives, for instance); and the serious-minded (assorted efforts to measure the student experience and academic strengths of campuses around the United States without sorting the results into ranked order, as with the National Survey of Student Engagement [NSSE] and Web-based report cards developed by groups such as the Association of Public and Land-grant Universities and the National Association of Independent Colleges and Universities).

With the exception of NSSE, which is aimed primarily at universities that wish to improve their own efforts to educate students, all these efforts cater, in one way or another, to the huge hunger of educational consumers for more and better information about educational quality and value for money. But whatever their strengths and credibility within academic circles, it is probably fair to say that none has had the sheer influence and popularity of the U.S. News rankings, their faults notwithstanding.

RANKING THE WORLD

That influence goes well beyond U.S. shores, which should perhaps be no surprise given that some of the same factors that made rankings ubiquitous in the United States have led to their rapid spread elsewhere. Often called “league tables,” the same term used in Europe for sports rankings, university rankings at the national level became common in the 1990s.¹⁰ The forty-plus national rankings that now exist around the world¹¹ can be found from Eastern Europe and the Middle East to Latin America and sub-Saharan Africa. By 2007, according to the region-by-region tally of Jamil Salmi, the World Bank’s Tertiary Education coordinator,¹² countries with rankings included Argentina, Australia, Brazil, Canada, Chile, China, Germany, Hong Kong, India, Italy, Japan, Kazakhstan, Korea, Malaysia, Mexico, the Netherlands, New Zealand, Nigeria, Pakistan, Peru, Poland, Portugal, Romania, Russia, Slovakia, Spain, Sweden, Switzerland, Thailand, Tunisia, Ukraine, the United Kingdom, and the United States.

The groups that prepare the rankings are as varied as the nations where they have emerged. Rankers include newspapers or magazines, accreditation organizations, universities themselves, and, increasingly, government agencies such as higher education ministries. Thus, country-level rankings in Britain, Germany, Canada, Italy, and Mexico are published by the Financial Times and the Sunday Times, Der Spiegel, Macleans, La Repubblica, and Reforma, respectively. Government and higher education organizations that rank institutions include Germany’s CHE (Center for Higher Education Development), India’s NAAC (National Assessment and Accreditation Council) and NBA (National Board of Accreditation), Turkey’s Council of Higher Education (YÖK) and TÜBİTAK (The Scientific and Technological Research Council of Turkey), and the Philippines’ PAASCU (Philippine Accrediting Association of Schools, Colleges and Universities). There are also many commercial guides, including Australia’s Good Universities Guide, Germany’s Bertelsmann Stiftung, and Canada’s Re$earch Infosource Inc.¹³

The factors included in these rankings vary, of course, as do the weightings, but they typically include indicators that should be familiar to any student of the U.S. News rankings. Institutions are usually ranked from highest to lowest based on a combination of quantitative and qualitative measures, including core statistics from universities about their research and teaching outcomes, along with surveys of students, peers, or outside analysts. Ellen Hazelkorn, director of the Higher Education Policy Research Unit at the Dublin Institute of Technology, notes that these national rankings, like those pioneered by U.S. News, began as a consumer information tool to provide students and parents with information, comparable across institutions, that was often not easily obtainable from universities themselves.

Over time, however, more evidence has accumulated suggesting that rankings are being used by a much wider variety of users, including government policymakers and industry officials. “Undoubtedly, part of the increasing credibility of league tables and rankings,” Hazelkorn writes, “derives from their simplicity and the fact that they are perceived as independent of the higher education sector or individual universities.”¹⁴ Despite profound worries over methodological shortcomings and misuse of rankings, just as in the United States, analysts such as Boston College’s Philip Altbach acknowledge that the assorted forms of rankings and league tables found around the world “serve a useful role” because of the spotlight they shine on “key aspects of academic achievement.”¹⁵

At the same Shanghai meeting of ranking experts where Canto-Sperber voiced her dismay at the way the École Normale Supérieure fared in global rankings, a number of other education officials detailed their own nations’ efforts to develop indicators of excellence, some quite different from one another. In Taiwan, for instance, Tamkang University published the country’s first national college ranking in 1993 with a twofold aim: understanding the overall academic performance of Taiwan’s 139 varied institutions and creating a useful self-improvement tool for Tamkang University itself. The creator of those rankings, Angela Yung-chi Hou, formerly a professor at the university’s Graduate Institute of Higher Education, notes that they were explicitly modeled after those created by U.S. News, with multiple criteria established—both quantitative and survey-based—and then weighted in order to rank each institution in one of eight categories. College rankings remain controversial in Taiwan, she concedes. But, she maintains, “it is expected that the quality of Taiwanese higher education could be improved through rankings.”¹⁶

In Romania, too, where no fewer than three rankings have been inaugurated, a group of researchers who have studied the classifications concludes that “ranking makes universities aware of their very weak and strong points and prepares them to measure up to competition under the circumstances of liberalizing the education and labor markets, once Romania accedes to the European Union.”¹⁷ Similarly, in the Republic of Kazakhstan, the Ministry of Education and Science directed its National Accreditation Center to conduct rankings of the nation’s universities beginning in 2006. The goals of the initiative, as in other nations, are to help provide decisionmaking tools to students and parents, government workers, employers, and international organizations; to promote competition between universities; and to encourage quality assurance within universities.¹⁸

Even in places where rankings have not yet penetrated, it seems, there is a hunger for them—at least among some. In Greece, two other researchers explained, rankings are strongly discouraged. Indeed, researchers are typically restricted from classifying and evaluating public universities, thanks in part to opposition from student groups and unions representing academic staff. Nevertheless, they suggested a number of possible approaches to ranking Greek universities, arguing that such measures could be useful to a number of groups: students seeking reliable information about campuses, universities seeking a diagnostic tool and measure of quality, the government, and society more generally.

While national-level university rankings have continued their seemingly relentless march around the world, probably the most influential comparisons between postsecondary research institutions are the global rankings that have emerged and become popular in the past fifteen years. The first international ranking was conducted in 1997 by Asiaweek magazine, but it was limited to universities in Asian nations.¹⁹ Five years later, in 2002, the Swiss Centre for Science and Technology Studies released its “champions league,” which ranked universities and other research institutions on the basis of their research journal publications.²⁰ However, the first closely watched worldwide ranking to appear on the global scene came the following year with publication of the Academic Ranking of World Universities, produced by Shanghai Jiao Tong University’s Institute of Higher Education, which some view as the most influential international ranking. First released in June 2003, the Shanghai rankings had their origins several years earlier in 1999. Shanghai Jiao Tong administrators, worried about the university’s decline from its once-exalted position in prerevolutionary days, began a series of planning meetings aimed at assessing where the university stood compared to others, particularly in the key area of research productivity.²¹ One particularly active participant, a professor of chemistry and chemical engineering named Nian Cai Liu, was drafted as the university’s first-ever director of strategic planning. His first task? Compiling a report comparing his university’s performance to that of others in China and elsewhere.

Liu’s timing was good. The rankings prototype, initially circulated privately to administrators and campus officials, came as China was embarking on a broad, ambitious, and expensive initiative to create a much-larger group of high-level research universities. Even well-known institutions had set ambitious targets to reach the much-coveted “world-class” status quickly: Peking University set its sights on 2016, for instance, while its crosstown rival Tsinghua University aimed for 2020. But without benchmarking against universities at home and abroad, determining just what was meant by “world-class” would have been difficult. That was where Liu’s work came in, and it was quickly expanded to include a much larger number of institutions around the world. That broader scope permitted the entire nation, not just Shanghai Jiao Tong, to figure out where it stood vis-à-vis universities around the world—and just how far it would have to go to close the gap.

Although the early rankings were aimed at a small audience, after Liu posted them online the demand—and controversy—soon became enormous. By his count, 4 million people have visited the Shanghai rankings Web site since 2003—an average of 2,000 per day.²² Liu and his research team pride themselves on having developed a transparent methodology, clearly explained on their Web site, that requires no complicated (or costly) survey research and instead relies on a range of publicly accessible indicators. His rationale for the Shanghai approach rests on several premises. First, while universities’ impact on economic development has become well established, “it is impossible to obtain internationally comparable indicators and data” of that contribution. Furthermore, while education is certainly “the basic function of any university,” differences across national systems make qualitative comparisons unworkable. The best approach, Liu concludes, is to look objectively at the research performance of universities, which he argues can be standardized and compared globally, providing a good indication of the relative standing of different institutions.

The Shanghai rankings begin by examining any university in the world whose faculty includes Nobel laureates; winners of the Fields Medal, granted every four years to a handful of elite mathematicians age forty and younger; and researchers who have published papers in Nature or Science, or whose work is frequently cited by others. It also looks at the overall number of academic papers at universities around the world indexed in the Science Citation Index Expanded (SCIE), the Social Science Citation Index (SSCI), and the Arts and Humanities Indices (ACHI). The World University Rankings researchers examine more than 2,000 universities and rank more than 1,000, posting the top 500 on their Web site.

The methodology itself bears some surface resemblance to the U.S. News ranking system in that it assigns varying weights to each of the factors measured. That said, unlike U.S. News, its research-intensive approach pays no attention to undergraduate-oriented measures such as student qualifications, class size, student retention, and graduation rate, nor to peer reputation. It assigns a weight of 10 percent for the number of an institution’s alumni who have won Nobel Prizes and Fields Medals since 1901; 20 percent for university staff winning those honors (with higher weights assigned to more recent winners); 20 percent for the number of highly cited researchers in a range of fields, including life sciences, physical sciences, medicine, engineering, and social sciences; 20 percent for articles published in Nature and Science within the past five years; 20 percent for the total number of articles by university faculty indexed in the SCIE, the SSCI, and the ACHI within the past year; and 10 percent for a size-adjustment measure that attempts to gauge per capita research performance by dividing each subtotal indicator by the number of full-time faculty and academic staff at each university.

Liu exhibits a refreshing lack of defensiveness when discussing his rankings, making the case for their strengths while cheerfully acknowledging their weaknesses. In some cases, he and his team have attempted to improve their methodology in response to critics. A case in point: Many detractors have complained about the Shanghai rankings’ bias toward science. After confirming that academics in the humanities typically have lower rates of publication than scientists, Liu doubled the weighting of articles that appear in the SSCI. The rankings are still viewed as heavily tilted toward science (“the easiest way to boost rankings is to kill the humanities,” one university rector told Hazelkorn). Still, Liu says he is glad to receive more suggestions for improvement. “We love those ideas,” he told the Chronicle of Higher Education. “We may not be able to implement all of them, but they’re great.”²³

The institutions at the very top of the Shanghai rankings come as no great surprise; the global pecking order closely resembles the one that exists within the United States. Thus, the top ten universities in 2008 were Harvard, Stanford, Berkeley, Cambridge, MIT, Caltech, Columbia, Princeton, Chicago, and Oxford. That all but two of the top ten institutions are U.S. universities reflects the massive dominance of the American research model on the world stage—all the more so, of course, when the scientific and technological accomplishments most closely associated with that model are so richly rewarded by a ranking methodology like Shanghai’s. At the same time, the rankings can provide bragging rights for those well beyond the inner circle—the University of Maastricht, for instance, is categorized in the band of universities between three hundred and four hundred (there are no numbered rankings beyond the top one hundred), along with institutions such as the University of Oregon and the University of Stuttgart.

One year after the debut of the Shanghai rankings, a second global effort to compare university systems was launched by a British publication, the Times Higher Education Supplement (THES). The World University Rankings quickly began vying with their Chinese rivals for influence on the world education scene. Views are mixed as to which ranking has garnered more attention from students, universities, and government policymakers, but it is broadly accepted that the THES assessment has far surpassed the Shanghai rankings in generating controversy.

The THES rankings were launched by John O’Leary, the onetime education editor of the London Times, who had previously overseen the national league tables produced by the Times, as well as a spinoff publication called The Times Good University Guide. In 2008, just before the magazine released its fifth annual rankings, O’Leary published an article that declared the ever-increasing importance of such global measures. “Particularly where research is concerned, Oxford and Cambridge are as likely to compare themselves with Harvard and Princeton as with other UK [institutions],” he wrote. “And governments all around the world have expressed an ambition to have at least one university among the international elite.” The league table exercise was conducted during its formative years in conjunction with the research firm Quacquarelli and Symonds (QS) and thus formally called the THE-QS World University Rankings (after a change of ownership the THES was renamed Times Higher Education [THE]). Its goal: to give a “rounded assessment” of top universities “at a time of unprecedented international mobility both by students and by academics.”²⁴

O’Leary’s reference to a “rounded assessment” is an unmistakable allusion to one of the key features differentiating the THE rankings from their Chinese counterpart. In contrast to the almost exclusive focus on research in the Shanghai rankings, the Times Higher Education methodology counts a much wider range of factors. Academic peer review is at the heart of the THE approach. It is based on about 9,400 responses over three years to an online survey distributed to academics worldwide, with the results weighted at 40 percent of the total²⁵—by far the largest factor (and considerably larger than the 25 percent that U.S. News devotes to its peer survey, a weighting that itself is often criticized for its disproportionate influence on the magazine’s college rankings).

The magazine’s rationale for such heavy use of a subjective reputational measure is that it avoids penalizing universities with nonscience strengths. Unlike the citations-per-professor score, the World University Rankings Web site explains, “the peer review component offers an even-handed perspective on the various broad subject areas—with institutional strengths in Arts & Humanities and Social Sciences able to contribute significantly to the overall view of an institution.” Somewhat undermining this defense, however, is the magazine’s acknowledgment in the very next sentence of its “frequently asked questions” that if it could identify “additional reliable measures of institutional quality,” it would likely reduce the weighting of peer review.²⁶ Still, it does make good on its promise of scrutinizing universities’ strengths in a variety of disciplines: in addition to its master rankings of universities worldwide, it also publishes tables each year, based on the peer review survey, that list the best institutions for arts and humanities, social sciences, life sciences and biomedicine, natural sciences, and engineering and information technology.

The second component of the THE methodology is “employer review,” weighted at 10 percent, which was introduced in 2005 and is based on a survey distributed to public and private sector employers around the world. In 2009 this score was based on about 3,300 responses over three years. An additional 20 percent is devoted to student-faculty ratio, based on the assumption that this measure serves as the best available proxy for an institution’s commitment to teaching. Next, THE uses citations per faculty member, weighted at 20 percent, to assess each university’s research prowess. Relying on Scopus, the largest database of abstracts and citations in the world, it draws on the most recent five years of citation data. The use of a per-professor measure is intended to control for institutional size. The measure doesn’t carry greater weight, the magazine explains, because it tends to be weighted toward research in scientific and technical fields. Finally, the magazine measures the percentage of international students and faculty at a university, on the grounds that this serves as a market test of an institution’s ability to attract brainpower in an ever-more globalized world. International students and faculty are each weighted at 5 percent for a total international measure of 10 percent.²⁷

Like the designers of the Shanghai rankings, the journalists and researchers behind the THE rankings say that they try hard to make the factors behind their evaluations easily accessible to the public. “It’s important that we get it right and that it is utterly transparent,” says Ann Mroz, the current editor of the THE, over lunch at a London restaurant.²⁸ She is also, like Shanghai Jiao Tong University’s Liu, quite happy to discuss misgivings about her publication’s rankings and entertain criticisms or suggestions for improvement. “I’m very keen for there to be a debate about it,” she says. “Any criticisms I’m quite happy to print. I would prefer that people came to us and there was some sort of debate about it and see whether maybe we have got a few things wrong. Until we discuss it, we’re never going to know.” Mroz herself says that she is “uncomfortable” with the faculty-student ratio, for instance. “It’s so crude. Does it tell you how good the teaching is?” She would like to use a better measure, she says—if one can be found.

Even as she acknowledges shortcomings in the rankings, however, Mroz firmly defends their usefulness. “If you’re talking about the students, what else do they have to go by?” she says. “There’s very little information, especially for foreign students. How are you going to try to compare a university of one country against another if you want to go abroad and study? I don’t think you should probably base your entire decision on the rankings, because that would be daft. You have to do a lot more research, but this is a good place to start.” Universities, too, rely on the rankings, she notes, whether as grist for marketing efforts when their standing rises or as a gauge of whether a prospective international partner has comparable worldwide standing.

While the THE is proud of its aspiration to create a more “holistic” assessment of universities than the Shanghai ranking, critics like Simon Marginson of the Centre for the Study of Higher Education at the University of Melbourne view its methodology as more problematic on a variety of grounds. They often fault the THE’s index for its high volatility, particularly vis-à-vis its relatively stable Shanghai counterpart. This was particularly true in 2007, when changes in data and methodology contributed to Stanford University’s drop from number 6 to number 19, the National University of Mexico’s plunge from 74 to 192, and the National University of Singapore’s decline from 19 to 33. “Think ‘yo-yo’ and you’ve about got it,” Marginson writes.²⁹

In some sense, of course, the charge of volatility punishes rankers for attempting to remedy their past sins. Large changes in an institution’s ranking typically come when the ranking organization—whether U.S. News or Times Higher Education—makes methodological changes in response to previous criticisms. In 2007, for example, THE prevented reviewers from assessing their own universities, changed its citation database to reduce what it called a “pronounced bias” toward U.S. institutions, and began using statistical normalization to control for outlier scores.³⁰ Still, to Marginson the frequent changes in the Times Higher rankings are highly suspect, as is the survey’s dismayingly low response rate (as little as 1 percent, he states).

Marginson also objects to what he terms a regional bias in which the survey’s “pool of responses was heavily weighted in favor of academic ‘peers’ from nations where the Times is well known, such as the UK, Australia, New England, and Malaysia.” Because of the survey’s “composition bias,” he argues, British institutions are vastly overrepresented. In 2007, “an amazing nineteen” U.K. universities placed in the top one hundred, compared to thirty-eight U.S. universities, a relatively modest showing given that fifty-four American universities placed in the top one hundred of the Shanghai rankings in the same year.³¹

In 2008, the very top of the THE and Shanghai lists overlapped significantly—seven of the ten universities were the same. But four British institutions placed in the THE top ten, while just two made it to the top of the Shanghai rankings. The Times Higher’s top ten in 2008 were Harvard; Yale; Cambridge; Oxford; Caltech; Imperial College, London; University College, London; Chicago; MIT; and Columbia. British institutions inched up still further in 2009, when they occupied four of the Times Higher’s top six slots. Meantime, U.S. superiority dwindled considerably: The number of North American universities in the Times Higher top one hundred dropped to thirty-six from forty-two the previous year.³² But the pecking order could well be shaken up even further in the future: several weeks after releasing its 2009 rankings, THE announced that it would end its partnership with QS. It said it would develop a brand-new ranking methodology in consultation with Thomson Reuters, a prominent data research firm, together with its academic advisory board and readers. “We acknowledge the criticism and now want to work with the sector to produce a legitimate and robust research tool for academics and university administrators,” Mroz said.³³ For its part, QS said it would continue to publish and circulate its own rankings, an indication that rankings proliferation shows no signs of letting up.³⁴

Even as the comparative strengths of each major global ranking continue to be debated—or as each is emphatically rejected by some university officials and students of higher education—their influence seems to be ever greater. In addition to consumer uses of the world rankings, they are increasingly an object of anxiety for universities—and a significant factor in their decision making. In a study commissioned by the OECD, the Dublin Institute of Technology’s Ellen Hazelkorn surveyed university leaders from 202 higher education institutions in forty-one countries, both new and old, teaching- and research-intensive. Rankings, she found, had become a hugely important factor in self-perception and decision making. “Despite the existence of 17,000 higher education institutions worldwide, there is now a near-obsession with the status and trajectory of the top 100,” she wrote in a summary of her findings.³⁵ Across a wide range of institutions, she told University World News, “there is enormous attention given to every league table that is published as well as its quality ranking. And they are taken seriously by students, government and especially by the media. Because of this, they have a huge influence on university reputations and thus they promote competition and influence policy-making.”³⁶ One manifestation of this intense interest in rankings: controversial efforts to “incentivize” administrators with cold hard cash to boost their institutions’ standing. For example, in Australia a number of vice chancellors have received salary bonuses predicated on their success in nudging their campuses up in the rankings.³⁷

Hazelkorn’s multination study found that 58 percent of respondents were unhappy with their current ranking, that 70 percent wanted to be in the top 10 percent of their national league table, that 71 percent wanted to be in the top 25 percent internationally, that 57 percent had a formal mechanism to review where they stood in the rankings, and that 57 percent thought that the willingness of other universities to form partnerships with them was influenced by their position in league tables and rankings. These perceptions aren’t just idle musings. Hazelkorn found that universities have often backed them up with concrete actions. Some go out of their way to hire more Nobel laureates, for example, given that this is a metric in the Shanghai rankings. More broadly, entire nations have paid special attention to revamping their university systems in the hope of achieving higher stature in the rankings. “Excellence initiatives in Germany, Russia, China and France are policy responses to rankings,” Hazelkorn writes. “The pace of higher education reform is likely to quicken in the belief that more elite, competitive and better institutions are equivalent to being higher ranked.”

Beyond survey data, it isn’t hard to find ways in which the siren call of rankings is heard far and wide. In India, both the THE and Shanghai rankings are scrutinized at the highest levels of government, according to Montek Singh Ahluwalia, who heads the nation’s Planning Commission. “We know about these lists. We look at them,” says Ahluwalia in an interview in his New Delhi office.³⁸ The Oxford graduate, whose formal title is deputy chairman of the commission, says the rankings serve in part to confirm India’s sense that some of its elite institutions—the Indian Institutes of Technology, the Indian Institute of Science, the Indian Institutes of Management, Jawaharlal Nehru University, and Delhi University—have earned a legitimate place among the world’s best. “If tomorrow they were to drop all of them, we would say it’s all biased and useless,” he quips.

At the same time, the rankings provide a gauge whereby a nation that urgently wants to increase both the quantity and quality of its institutions can measure its progress. “Assuming we have about four or something like that [among the several hundred top institutions in the global rankings], everybody in India thinks that in ten years we should have at least ten and hopefully twenty,” he says. “That’s not easy to do, because you don’t create a good university in ten years. But maybe you can upgrade some of the existing ones and so on.” Improvement at the elite levels is even more daunting than improving mass access to higher education, he says. “Taking the number from four to ten will require some of the brightest brains in government to do some very innovative thinking on reforms for universities.”

In addition to serving as a broad measure of quality for nations intent on improving their international standing, rankings can also act as a great leveler. In the best-case scenario, they allow individual institutions or programs to prove their worth against better-established competitors. This can be seen particularly clearly in the case of business schools, which were early adapters to globalization.

As goods, services, and people started to move ever more freely across borders, a far-reaching market for globally literate MBAs soon emerged. Business schools began to market themselves aggressively to foreign students, to start branch campuses, and to forge alliances with their counterparts in other nations. The results of all this internationalization were striking. To take a few examples: Between 2004 and 2008, the number of U.S. MBA applicants sending scores on the Graduate Management Admissions Test to non-U.S. programs increased by 35 percent. At INSEAD in Fontainebleau, no more than one in ten faculty members comes from any single country. Madrid’s IE Business School has recruitment offices in Berlin, Dubai, Lisbon, and Singapore. And Alpina, a dairy and beverage manufacturer in Colombia, extended fifteen job offers to MBA grads from IE in 2008 and planned to begin recruiting at INSEAD and the London Business School.³⁹

This rapid globalization was accompanied, and perhaps hastened, by a slew of business school rankings. Like comprehensive university rankings, these began at the national level before expanding globally. BusinessWeek, for example, pioneered U.S. MBA rankings in 1988 and was later joined by Forbes and the Wall Street Journal. Now BusinessWeek has added to its offerings an annual list of “Best International B-Schools,” while the Economist and the Financial Times have developed their own high-profile global business school rankings. The methodology of each ranking differs, but all include some measurement of alumni salaries, postgraduation career success, or both. One analysis found a fairly high correlation between almost all the MBA rankings.⁴⁰

Like other rankings, the business school league tables have attracted critics and controversy. However, by providing a neutral yardstick that measures schools’ effectiveness on key measures that matter to students and employers, the rankings have the power to confer instant legitimacy on relatively new players on the global B-school scene. China’s CEIBS (China Europe International Business School), for instance, is only fifteen years old. But it has thrived in a short time and now touts its high rankings on its Web site: number eight worldwide in the 2009 Financial Times global MBA rankings, number four in the Forbes 2009 list of top non-U.S. business schools, number one in BusinessWeek China’s ranking, and so on.⁴¹ By offering an external measure of CEIBS’s success, these international business school rankings illustrate a point that economics columnist Robert Samuelson has made in the context of U.S. college rankings:⁴² there is a strong case to be made that rankings have the potential to radically democratize the entrenched academic pecking order—on the global as well as on the national scene.

Rankings are also being used more generally by students or national governments as the equivalent of the Good Housekeeping Seal of Approval. They can give educational consumers a sense of which overseas institutions are likely to offer value for money. Similarly, they can inform governments about whether their own policies are well considered or their scholarship funds well spent. The Mongolian government has weighed a policy that would give study-abroad funding only to students admitted to a university that appears in one of the global rankings.⁴³ In the Netherlands, an immigration-reform proposal aimed at attracting more skilled migrants would restrict visas to all but graduates of universities ranked in the two top tiers of global league tables.

But if government quality-assurance efforts, consumer advice, and performance incentives for campus officials represent a largely benign aspect of global rankings, another is the ignominy—and real-life consequences—that can fall upon individuals and institutions that are viewed as not measuring up. The most oft-cited case of this phenomenon came in 2005 at the University of Malaya (UM). One year earlier, the Malaysian university drew accolades when it was ranked eighty-ninth in the world in the inaugural Times Higher Education Supplement rankings. So important an accomplishment was this for the university, and for a nation bent on creating a knowledge economy, that the vice chancellor ordered banners reading “UM a World’s Top 100 University” and had them hung around that city (and, Marginson notes, “on the edge of the campus facing the main freeway to the airport where every foreign visitor to Malaysia would see it”⁴⁴).

The next year, however, the results of the Times’ two reputational surveys were less favorable to the university. Compounding matters was the discovery and correction of an error in the classification of foreign students at UM, which served to further lower the university’s rating. The result? A drop from 89 to 169 in the THE survey and widespread calls for a royal commission of inquiry into the unfortunate episode. “A Shocking Global Slide,” read one headline.⁴⁵ “Crisis in Malaysia’s Public Universities?”⁴⁶ inquired another. Within a few months, the vice chancellor, who had been vilified in the media, was effectively fired. “Though apparently extreme, this reaction is not uncommon in university systems around the world,” writes World Bank higher education expert Jamil Salmi.⁴⁷ It has certainly remained common in Malaysia. Just a few years later, when none of the country’s universities placed in the THE top 200 in 2008, Lim Kit Siang, leader of the opposition Democratic Action Party, gave a speech to party supporters, declaring that Malaysia “is losing out in the unrelenting battle for international competitiveness,” not only worldwide but vis-à-vis regional competitors such as Thailand, Indonesia, and the Philippines. He complained bitterly about the showing of the Universiti Sains Malaysia, which the government has singled out for cultivation as a world-class institution, calling its 313th showing “sad and pathetic.” The rankings, the opposition leader concluded, “should be a wake-up call to the Higher Education Minister and the cabinet of the advanced crisis of higher education in Malaysia.”⁴⁸

In Britain, too, analysts reacted with similar alarm, albeit in less heated language, when the 2008 THE rankings showed a drop for many U.K. universities. Both Oxford and Cambridge slipped slightly in the top ten pecking order, and overall twenty-two of the twenty-nine British universities in the top two hundred fell in the rankings. While Britain still had more highly ranked universities than in any nation outside the United States (which held thirty-seven of the top one hundred slots), the Daily Telegraph noted that universities from thirty-three different countries made it into the top two hundred, an increase from twenty-eight in 2007.⁴⁹ Though relatively modest, these shifts were enough to prompt a follow-up article headlined “Without Investment Our Top Universities Will Fall Behind Global Competition.”⁵⁰

Its author, Wendy Piatt, head of the Russell Group, a consortium of twenty elite research universities, noted that the United States invests more than twice what Great Britain does in higher education as a proportion of gross domestic product. What’s more, she said, Chinese universities “have been steadily climbing up international league tables” and are on the verge of overtaking their British counterparts in faculty publication of research papers. Adding to the competitive environment, “closer to home, France and Germany are both pumping millions into their leading research universities,” Piatt wrote. Her conclusion, perhaps unsurprisingly, was that notwithstanding concerns about the rankings’ accuracy, their message should nevertheless be heeded as a sign that increased investment in U.K. universities is imperative.

While the range of responses to the global rankings from universities and policymakers shows their unmistakable influence, many detractors believe the rankings are unworthy of any kind of response. To begin with, there are the familiar and highly specific criticisms. These include, on the one hand, the frequently denounced bias toward science in the Shanghai rankings, coupled with the incentives the Shanghai approach gives universities to engage in questionable chasing of Nobel-winning professors whose work may or may not be of sufficiently recent vintage to add meaningfully to the institution’s intellectual firepower. They comprise on the other hand, of course, the excessive reliance on peer reputation, low sample size, volatility, and English-speaking bias of the THE rankings.

But to some, the very notion of attempting to determine how an entire university stacks up against others is an exercise that has little meaning as an instrument of national policy. Ross Williams and Nina Van Dyke of the Melbourne Institute of Applied Economic and Social Research at the University of Melbourne argue that evaluating individual disciplines across universities makes more sense. After all, for many students and researchers making decisions about where to apply and study or work, and for government agencies seeking to fund excellent research, the quality of specific departments and disciplines is more important than the university’s overall standing. Broad institutional rankings can make universities look worse—or better—than they really are. Their survey of a range of disciplines at thirty-nine Australian universities—arts and humanities, business and economics, education, engineering, law, medicine, and science—found that in twenty-three cases, a discipline at a particular institution was rated among the top one hundred in the world.

With universities in Australia and beyond under considerable pressure to place ever higher in the world rankings, Williams and Van Dyke argue that a focus on disciplines would, instead, encourage more specialization in fields in which a university may have a particular strength or comparative advantage. “At whole-of-institution level,” they write, “it is not reasonable to expect, under current resourcing levels, more than three or four Australian universities to be in the top one hundred in the world, but it is feasible for many more Australian universities to be in the top one hundred in individual disciplines. . . . A system in which each Australian university was recognized internationally for some activity, including teaching, would be preferable to the current situation.”⁵¹ Indeed, beginning in February 2007, the designers of the Shanghai rankings began ranking universities by broad subject fields in addition to overall quality. Still, in many places, the obsession with making it to the top remains, however unrealistic the goal may be. Another Australian university official, Tony Sheil of Griffith University’s Office for Research, argues that it would be prohibitively expensive for Australia and other small nations to make the kind of investments necessary to catapult a university into the top ten or twenty of the global research rankings, notwithstanding the expressed desire of several recent Australian federal education ministers to reach that goal. (The current minister has moved in a different direction, focusing on creating a world-class university system nationwide rather than on rising to the top of the rankings.)⁵²

More broadly, just as many critics of the U.S. News rankings question the premise that institutions with widely varying missions can be meaningfully evaluated by outsiders, some academics protest the notion that global league tables capture the essence of institutional quality. While the aspiration to be world-class seems to be at the top of every university’s to-do list, this argument goes, worldwide rankings are unavoidably a zero-sum game that implies excellence is found only at the heights of the league tables. As Franz Van Vught of the European Center for Strategic Management of Universities argued at an OECD conference on measuring quality in higher education, if just 3 percent of the world’s 17,000 universities are world-class as measured by rankings, surely the rest cannot have utterly failed.⁵³

France has been a particular hotbed of rankings discontent. Even as Prime Minister Nicolas Sarkozy has pushed to shake up the nation’s moribund university system and create a lavishly funded group of world-class research universities, academics and some government officials have simultaneously expressed vocal discontent with global rankings. They complain among other things that the Shanghai rankings favor universities in English-speaking countries, don’t take institutional size into account, and fail to measure the quality of teaching.⁵⁴ Moreover, some fear that student qualifications have been lost in the global rankings frenzy. Canto-Sperber of the École Normale Supérieure (ENS) argued before her Shanghai audience that ultra-selective colleges such as France’s grandes écoles don’t get rightful credit for the rigors students must undergo before they even begin their studies.

After graduation from lycee, she explained, the most brilliant students in France enroll in rigorous classes préparatoires before they can attempt entry to ENS. In these courses, “the competition between students is such that one who succeeds by the end of classes préparatoires would have typically studied for twelve to sixteen hours a day without holiday for two or three years.” Among the graduates of these courses, she noted, “only the top few are allowed to enter the École Normale Supérieure,” which stands as the most selective educational institution in France. Notwithstanding this intense selection process, ENS took the seventy-third slot in the Shanghai ratings in 2008. Institutions such as the University of Paris VI or the University of Paris XI were ranked higher despite, Canto-Sperber observed pointedly, having “completely nonexistent selection procedures (i.e., everyone who applies is admitted).” Ironically, other rankings that take student qualifications into account—notably those produced by U.S. News—are frequently denounced for focusing excessively on inputs rather than outputs. Still, Cantor-Sperber insisted, “the quality of a university is based on its own procedures of student selection. Therefore, the criterion of student selection has to be considered for the evaluation of universities.”

Given all this discontent, it is little wonder that some of those deeply dismayed by the shortcomings of existing rankings have begun to develop what are essentially counter-rankings. In India, where relatively few universities have merited inclusion in the global rankings, the University Grants Commission in 2009 proposed its own ranking system, to be called the India Education Index, which would grade Indian institutions with respect to their international peers.⁵⁵ In France, another prestigious grande école, Mines Paris Tech, released its second Professional Rankings of World Universities in October 2008, leading to a memorably tautological headline in the online publication University World News: “French Do Well in French World Rankings.” Perhaps inevitably, however, this alternative ranking itself has been criticized for being one-dimensional: its sole criterion for sorting the 350 institutions surveyed is the number of graduates serving as CEO or the equivalent in companies listed in Fortune magazine’s Fortune Global 500. Using this measure, five French universities placed in the top twenty alone, including two in the top ten. By contrast, just three French universities appeared in the entire top one hundred Shanghai slots, and only two were included in the top one hundred of the Times Higher Education league table.⁵⁶

Despite Sarkozy’s determination to place some French universities in the world’s top tier by 2012, France’s disdain for the current rankings (Valérie Pécresse, the nation’s higher education minister, once said that the problem with rankings was that they existed) has extended to efforts to create a Europe-wide alternative. As France took over the presidency of the EU, it convened a Paris conference in late 2008 to explore a range of international comparative measures that participants hoped might do better justice to the strengths in, say, teaching and innovation, that tend to be underrecognized by existing measures. By the summer of 2009, the European Union announced that it would begin developing a new “multidimensional global university ranking.” Mostly focused on Europe, the goal of the new assessment, still in the exploratory stage, is to move beyond research in hard sciences to include humanities and social sciences, as well as teaching quality and “community outreach.”

BETTER INFORMATION

Is there really a better way to rank universities? In addition to nationalist efforts like those of India and France, which invariably generate suspicions of chauvinistic intent, other attempts have been made to zero in on specialized aspects of the higher education enterprise that are overlooked by conventional rankings. In Spain, for instance, the “Webometrics Ranking of World Universities” was launched in 2004 to measure universities’ Web-based activities—specifically the “volume, visibility, and impact of the Web pages published by universities.”⁵⁷ Developed by the Laboratoria de Cybermetrics, a division of the National Research Council, Spain’s largest public research body, these rankings place special emphasis on Web-based publication of scientific output, including refereed papers, conference contributions, theses, and reports, as well as courseware, digital libraries, databases, personal Web pages, and more.

The goal of the effort, which was first launched in 2004 and is now updated every six months, is to promote electronic publication by universities, and in particular to encourage university administrators to do more Web-based dissemination if their institution ranks poorly. While students shouldn’t use these rankings as the sole criteria for choosing a university, the Webometrics creators say, a top position among the 17,000 higher education institutions worldwide⁵⁸ that are listed in the survey tells candidates that “the institution has a policy that encourages new technologies and has resources for their adoption.” Despite this disclaimer, analysts such as Richard Holmes of the MARA Institute of Technology in Malaysia note that important aspects of a university’s quality—teaching excellence or book publication, for instance—are not captured by this Web-centric indicator.

Another alternative measure that has attracted considerable attention in the past few years as a kinder, gentler form of evaluation comes from the Center for Higher Education Development (CHE), a German higher education reform think tank. In collaboration with a media organization (once Stern, later Die Zeit), the organization surveys 200,000 students and 15,000 professors at more than 250 universities,⁵⁹ mostly in Germany but also in Austria, Switzerland, the Netherlands, and recently Italy. The rankings include a range of quantitative measures, including student-professor ratio, average length of studies, failure rates, number of PhDs, and research productivity and funding; however, about two-thirds of its indicators are based on the survey questions. Students are asked about their experiences and overall satisfaction on their campus. Faculty are asked about their “insider’s pick”—which three institutions in their own field they would recommend to their own son or daughter.

After all these data are collected, they are not weighted or in any way used to create an ordinal ranking of participating universities. That would be far too simplistic, say the survey founders. “There simply is no ‘best higher education institution,’ not in one subject and certainly not in all subjects,” the organization’s Web site declares. “For example, a university may indeed be a leader in the field of research, but the equipment it offers its students may be miserable, or it may be strong in German Studies, but poor in Economics and Business Administration. Instead of crowning some presumed overall winner, we offer a multidimensional ranking.”⁶⁰ Along with avoiding comparisons between entire institutions, even within disciplines CHE stays away from numerical rankings and simply categorizes a given department as either in the top third, middle third, or bottom third compared to its peers. It also gives its rankings a strong element of consumer empowerment by permitting individual users to create their own combinations of the indicators they consider most important, and then order institutions accordingly.⁶¹

Still, while the CHE approach to rankings may appeal to certain constituencies, by design it sidesteps what may be a natural desire by policymakers and consumers alike to make judgments about which institutions are most effective overall—which ones are the best. Nor does CHE address the increasing interest in value-added assessment in higher education, which aims to assess how good a job universities do, not just in garnering research laurels—and Nobel laureates—but in passing on knowledge to their students.

One of the most closely watched experiments in ranking and assessment is attempting to do just that. This new initiative is known as AHELO, the Assessment of Higher Education Learning Outcomes. Its origins trace back to June 2006, when a group of OECD education ministers met in Athens and concluded that as higher education expanded massively, it was important to do more to measure quality as well as quantity. The resulting project is premised on the notion that students and employers are seeking better information with which to make choices about either attending universities or hiring their graduates; that universities and professors need to know more about the strengths and weaknesses of different institutions; and that policymakers need a better sense of the impact of their decisions on university quality. The OECD’s response has been to create an instrument intended to be valid “for all cultures and languages.”⁶² It is explicitly intended not to be a ranking in the Shanghai or Times Higher Education vein, but instead to focus on teaching and learning—not on inputs but on outputs and value added.

The initial design of the test focuses on four “strands” intended to reflect some of the crucial aspects of higher education. The first “generic skills” component attempts to measure students’ abilities in such areas as analytical reasoning, written communication, ideas generation, and application of theory to practice. Such abilities are not explicitly linked to a particular course of study but are nevertheless vital characteristics of what students should be learning on campus. “The point is that the simple acquisition of knowledge is not enough to count as an education,” as the OECD puts it.⁶³

The model for this part of the OECD’s new outcomes project is a test known as the Collegiate Learning Assessment, or CLA, which was developed in the United States by the Council for Aid to Education, an offshoot of RAND, the social science research organization. Since 2000 it has been used in hundreds of American colleges and universities to measure the kinds of skills all undergraduates should acquire, regardless of major. Researchers administer a computer-based exam, including essay questions, to a sample of students, typically a group of freshmen and a group of seniors. By controlling for the qualification of incoming students (as measured by their scores on the SAT or ACT), the CLA staff arrives at a value-added measure that attempts to show just how much students at a given university tend to improve their writing and analytical skills during their time on campus. Their research methodology has been controversial among some of their higher education colleagues, but the CLA’s designers are highly regarded social scientists, and the question they attempt to answer—what do students really learn on campus?—has rarely been addressed in such a systematic way. For the AHELO project, the College Learning Assessment will be adapted to fit an international range of universities. As in the United States, the notion is to ask nonspecialized questions that undergraduates in any field of study can answer. The inaugural participants in this part of the new test will certainly test its cross-national aspirations: they include Finland, Korea, Mexico, and Norway.

The second component of the OECD’s exam is designed to acknowledge that universities most often define their missions in terms of subject-specific knowledge, not generic skills. “Students and faculty would be astonished if an assessment left out the very reason they are in higher education,” the OECD says. Thus, this strand tests what students have learned within their own disciplines. At the feasibility-study stage, the two areas tested will be engineering and economics, with the expectation that more disciplines will be added if the project goes to scale. AHELO’s designers are quick to note that subject knowledge isn’t just about understanding facts but about putting that content knowledge to use, “often in novel circumstances.” Australia, Japan, and Sweden will take part in the inaugural round of engineering testing, while the economics tests will be administered in Italy, the Netherlands, Mexico, and the Flemish-speaking parts of Belgium. As OECD officials prepared to begin testing during the 2010–11 academic year, they announced a crucial development: the United States will take part in AHELO, with institutions in four states–Connecticut, Massachusetts, Missouri, and Pennsylvania–administering the generic skills test.

The third of AHELO’s four sections rests on the notion that student learning outcomes have to be understood in context, from students’ backgrounds to the characteristics of the universities they attend to what employers expect of them. To better understand such variables, this “context” section examines campus characteristics, such as total enrollment and male-female ratio; educational practices and quality, including student-faculty interaction, emphasis on hands-on learning, and level of academic rigor; what the OECD terms “psycho-social and cultural attributes,” from what society expects of postsecondary institutions to students’ career expectations; and various outcomes, both in behavior and attitudes, from degree completion and progress into the job market or graduate school to student satisfaction, self-confidence, and self-reported learning gains. Researchers will gather data for all these measures by examining public statistics, reviewing earlier research, and surveying students, professors, and university administrators. Eventually, they hope to develop alumni and employer surveys if and when a full-blown AHELO assessment is developed.

The fourth and final strand is intended to zero in on the value-added component of higher education, one that is increasingly scrutinized in the era of measurement and accountability. AHELO researchers pose an important question: When a top student enters a university and exits with similar levels of accomplishment, how much has the institution really done with the “raw material” that walked through its doors? By contrast, when a student enters with a B average and leaves campus with an A average, a case can be made that the university has performed a more valuable pedagogical role. “What a student brings to a degree programme and what he or she leaves with are a powerful indicator of teaching quality, availability of resources, and the capacity of students to learn,” the OECD says. OECD researchers acknowledge that consumers of league tables care a lot about absolute measures of quality, not just relative growth. Nevertheless, they say, a comprehensive assessment such as AHELO should offer both “bottom line” as well as value-added measures to provide a full picture of how well universities are educating their students. Unlike the other three strands, however, the value-added measure is not yet being carried out even in an experimental way. Given the complexity of the task, researchers say, there is not enough time to develop an appropriate measurement tool during the initial AHELO study. Instead, they are considering possible methodologies, drawing on similar work being done by the OECD at the secondary school level.

Indeed, the implicit model for the OECD’s new international effort is a respected assessment known as PISA—the Program for International Student Assessment—which was developed by the organization in 2000 and is administered to fifteen-year-olds in most OECD countries (and in some nonmember nations) to gauge the academic progress of students in one country vis-à-vis their peers in other industrialized nations. While it is not without critics, PISA provides an easily understandable gauge of global student achievement at the secondary school level. Indeed, as the AHELO project was getting under way, an OECD paper describing the new effort was titled “PISA for Higher Education.” Despite the attention received by existing national and international university rankings, they may distort resource allocation and thus give short shrift to teaching and learning, the October 2006 background memo observed. Instead, it declared, “a direct assessment of the learning outcomes of higher education could provide governments with a powerful instrument to judge the effectiveness and international competitiveness of their higher education institutions, systems and policies in the light of other countries’ performance, in ways that better reflect the multiple aims and contributions of tertiary education to society.”⁶⁴

However, drawing a direct parallel to PISA has proven contentious. Given the controversy surrounding rankings, it is perhaps unsurprising that within a couple of years, OECD officials were insisting that the word not be applied to their still in gestation postsecondary effort. “AHELO is not PISA for higher education,” declared Barbara Ischinger, director of OECD’s education division, at the opening of a major OECD conference on assessing quality in higher education. In a similar vein, OECD officials maintain that their current efforts to test out the new measure in a range of nations (they will assess students at some ten postsecondary institutions in three or four countries for each of the four strands) should not be considered a pilot but merely a “feasibility” study. Any next steps, they say, will be determined only on the basis of the outcomes of their exploratory work. Still, OECD representatives acknowledge that the project is being conducted “with an eye to the possible creation of a full-scale AHELO upon its completion,”⁶⁵ and some of those involved in the process say it is a virtual certainty that it will go forward.

Together with the contentious PISA analogy, however, a related concern quickly surfaced. Influential American higher education officials initially expressed deep misgivings about the OECD’s initial efforts, arguing that efforts to create a global postsecondary measurement and accountability system were inherently problematic, given the difficulty of finding a measurement instrument suitable for the wide variety of institutions involved in the effort. After all, huge controversy had already surrounded the Secretary of Education’s Commission on the Future of Higher Education, which endorsed the use of outcome-measurement tools such as the Collegiate Learning Assessment, opposed by some university officials as overly simplistic. “The conversations in the last year have underscored for many folks both the importance of addressing issues of student learning outcomes and the difficulty of finding a common instrument for measuring them in the United States,” Terry Hartle, a senior official at the American Council on Education, the umbrella lobbying group for U.S. colleges and universities, told Inside Higher Ed. “If we haven’t been able to figure out how to do this in the United States, it’s impossible for me to imagine a method or standard that would work equally well for Holyoke Community College, MIT, and the Sorbonne.”⁶⁶

But defenders of the new approach argue that while international comparisons may be challenging, there is nothing particularly new or objectionable about them. For instance, rankings expert Alex Usher, former head of the Canada office of the Educational Policy Institute, points to the example of the International Adult Literacy Survey, which is administered around the world, including in the United States, without incident. What’s more, reiterates Andreas Schleicher, head of education research for the OECD and a key designer of the PISA testing regime, the breakthrough of AHELO is that it will shift the rankings conversation in a crucial new direction. “Rather than assuming that because a university spends more it must be better, or using other proxy measures of quality, we will look at learning outcomes,” he says.⁶⁷ Initially, of course, the small number of universities taking part in the test in each country means that only national-level results are likely to be available. Ultimately, however, with sufficiently widespread participation, the OECD would be able to publish its own cross-national league tables—with the important difference, champions of the new approach say, that they would be based on results rather than on reputation. “We will not be reflecting a university’s history, but asking: what is a global employer looking for?” Schleicher says. More important still, such measures have the potential to help students, governments, and universities themselves focus on more meaningful measures of quality when making educational decisions.

One prominent convert to the usefulness of ranking is Jamie Merisotis of the Lumina Foundation. When he first began investigating college rankings a few years back as then president of the Institute for Higher Education Policy in Washington, DC, he took the view, widely shared in the U.S. higher education community, that rankings are “fundamentally flawed” because of dubious methodology and a host of other shortcomings. But he has since come to be a leading proponent of two complementary views: that rankings are a fact of life, and that they serve a useful function. Thus, not only is resistance futile—not his exact words, to be sure—but a more productive approach is to improve the quality of educational data, among other factors used in league tables, in order to make rankings better.⁶⁸

“The reason rankings are popular is that they actually serve a purpose,” Merisotis says. Until the emergence of rankings, he argues, existing quality-assurance mechanisms left a glaring gap. In the United States, government regulations focused largely on such matters as whether a college or university met the minimum standards to be eligible for federal student aid. Accreditation has been a purely peer-based quality recognition process, with little or no transparency to the outside world. Rankings, he says, “are basically reflecting the market’s desire for more information that the other two aren’t bringing forward.”

This insight applies not just to national rankings but to the burgeoning multicountry and global assessments as well. “What you’re seeing is increasing recognition of the fact that people who have college educations are in a global employment market,” Merisotis says. “Those individuals who are in that global employment market are being trained in a variety of institutions, for which we need to have a better understanding of which ones are good and which ones are less good.”

Merisotis, who frequently joins international partners to convene global gatherings to discuss rankings, is cautiously optimistic that experiments by the OECD and others may produce more fine-tuned university assessments. “My view is that the state of the art right now is that there are no really good global rankings,” he says. “But that doesn’t mean there won’t be.” While AHELO may or may not succeed, he makes a persuasive case that the effort to collect data and assess learning outcomes across nations, languages, and cultures is invaluable. Merisotis is also a fan of a decade-old European assessment effort known as Tuning, a faculty-led process that attempts to focus universities on establishing clear learning expectations for students in every field. Indeed, the Lumina Foundation is now working with state and university officials in Indiana, Minnesota, and Utah on a yearlong effort to attempt Tuning reforms in U.S. institutions.

Still another indication of where rankings may go in the future can be seen in the European Union’s new “multidimensional” assessment, with its emphasis not only on research but also on teaching quality and areas such as community outreach. The rankings are expected to focus on Europe but to have worldwide reach.⁶⁹ Rankings detractor Simon Marginson believes that the EU’s new effort will “change the game by lifting the quality of holistic rankings” such as those produced by Times Higher Education. That said, he is still dismayed by the continuing problem of “arbitrary weightings” and other shortcomings of existing global rankings. He predicts that the Shanghai and Times Higher rankings won’t go away, but that they are likely to be joined by a growing number of “custom-built” assessments that are disaggregated by field, employment rates, and the like. Thus, the department-by-department rankings that Marginson and others advocate will likely gain popularity even as universities, policymakers, and students continue to pay attention to the broad institutional rankings that have proven so controversial.

This cacophony of assessments, while likely at times to be hard to compare and contrast, is surely something to be embraced. For many years, relatively few external measures of university performance were available, particularly measures easily understood by consumers and policymakers. Rankings have emerged to fill that void. They are no doubt going to multiply and become more sophisticated as they mature. And they are no doubt going to remain a fixture on the higher education scene, emblematic of a world in which apples-to-apples, cross-border comparisons of educational quality are ever-more necessary.

In a relatively short period of time, a remarkably wide variety of rankings has spread and evolved around the world, from the national to the global, from reputation-based to research-based, from subject-specific to university-wide, from Web-oriented to multidimensional and unweighted, from the Princeton Reviews’s annual list of “top party schools” to the OECD’s sophisticated amalgam of value-added approaches. The interest such efforts has attracted in far-flung locales is reflected in the meeting places periodically chosen by a geographically diverse group of global ranking experts, who have roamed from Washington DC to Berlin to Shanghai and then, in 2009, to Kazakhstan.

Still, rankings remain highly contentious nearly everywhere, from the United States to Europe to Asia and beyond. For some critics, the very enterprise of sorting colleges in rank order is suspect. Uwe Brandenburg, project manager at Germany’s Center for Higher Education Development, quotes Einstein to make the point: “Not everything that can be counted, counts, and not everything that counts can be counted.”⁷⁰

Nevertheless, Brandenburg acknowledges that rankings can provide useful transparency so long as they are used in combination with other factors. A recent study of rankings in four countries, conducted by the Institute for Higher Education Policy, found that despite some potentially negative effects, such as encouraging a focus on elite research institutions, rankings had a useful impact on how universities make decisions, including more data-based assessment of success.

It is easy to see why, in a global market, students, universities, and governments have a growing need for better information about the comparative effectiveness of postsecondary institutions. But they need the right kind of information. And there are numerous barriers to providing it: many rankings are imperfect, to say the least, and refinements both large and small are badly needed. Even some of the most promising efforts, like the OECD’s AHELO project, may prove hard to implement—and also have the central flaw of paying zero attention to research. While human capital is an important output of universities, so is the research on which so much innovation and economic growth is dependent. Striking the right balance in assessing universities will be very important: one could imagine a ranking that takes three parts Shanghai, adds five parts AHELO, then throws in two parts Times Higher Education, to create a mixture that is useful to students and policymakers alike.

As more sophisticated rankings are developed, what are the chances that they will be widely implemented? The United States is a good proving ground for this question: it is the nation where college rankings were pioneered and remain hugely influential and, at the same time, a country where rankings routinely encounter withering criticism and steep resistance from academics and university leaders. Even as better measures of student-learning outcomes are developed, then, barriers to their introduction at American colleges and universities will likely remain high. What should state and federal policymakers do to help develop better rankings or to ensure that universities cooperate?

At the federal level, probably nothing. In some nations—Kazakhstan, perhaps—universities would likely have little choice but to participate in rankings sponsored and mandated by the government. But in the United States, with no tradition of centralized federal control of education, top-down efforts to mandate participation in either international or domestic assessments are unlikely to be successful. To be sure, for twenty-five years, the U.S. Higher Education Act—the standard-setter for accreditation rules, and thus the ultimate gatekeeper for billions in federal financial aid, has required colleges to report evidence of student academic achievement. However, efforts to satisfy this mandate through the accreditation system have taken what one analyst calls “a kaleidoscopic” variety of forms, and despite recent improvements, many have been ineffectual as instruments of accountability either within or outside universities.⁷¹ Even rather tame suggestions that colleges should use uniform measures of student learning outcomes have met with alarm. The Spellings Commission’s support for the CLA and NSSE was quickly, and perhaps mischievously, said to mean that it was advocating a federal No Child Left Behind Act for higher education—anathema to the academy and likely to be a political nonstarter.

States might have better luck introducing standardized measures of learning outcomes, and even using them for accountability purposes. After all, while U.S. higher education is remarkably diverse, high percentages of students are enrolled in state colleges and universities that are subject to the scrutiny, sometimes intense, of state lawmakers. Texas probably offers the best example of a successful state-mandated accountability system, which in many ways can be viewed as an extension of its widely imitated elementary and secondary accountability framework. Beginning with an executive order from Governor Rick Perry in 2004, the state has developed a comprehensive accountability system that requires state universities, among other things, to participate in the CLA and NSSE and to make the results public. Early results are positive, and if any additional state is likely to jump on the AHELO bandwagon, it would surely be Texas. Elsewhere, however, it is by no means clear that enough states share Texas’s results-oriented education culture to allow for the meaningful national comparisons that are crucial to assessing universities’ relative educational effectiveness.

It may well be that the best route toward widespread participation in the new generation of national and global rankings and assessments is the Consumer Reports model. No federal regulations or state laws require dishwasher manufacturers or automakers to submit their wares for testing and inspection by the widely consulted magazine. They do so anyway, knowing that the results will be pored over and referenced for years by potential customers, because the only thing worse than such comparative scrutiny would be no scrutiny at all. Indeed, in most of the nations where rankings have become increasingly pervasive and influential, universities face no government mandates to participate—they comply voluntarily with journalistic efforts such as those of Japan’s Ashahi Shimbun or Canada’s Maclean’s. Similarly, most American universities do respond, albeit reluctantly, to the U.S. News survey.

But it will be more challenging to persuade them to participate widely in more sophisticated—and publicly available—measures of their educational effectiveness, either nationally or internationally. So far, U.S. universities have shown a greater appetite for self-assessment than for transparency. Most of the colleges that participate in the CLA and NSSE prefer to keep their results private, using them for “self-study” rather than to inform potential students, parents, taxpayers, and lawmakers about how much students are actually learning.

One promising effort that moves colleges toward increased openness is the Voluntary System of Accountability, or VSA, which grew out of the increased pressure the Spellings Commission put on the higher education world to assess student outcomes. A joint effort of the American Association of State Colleges and Universities and the Association of Public and Land-grant Universities, which together grant 70 percent of the bachelor’s degrees awarded in the United States,⁷² the VSA requires participating schools to choose one of several learning assessment tests. Following a period of experimentation, the results must be published, along with other data, such as graduation rates, in a College Portrait that is uniform across institutions. It remains to be seen how effective this initiative will be, and whether a critical mass of universities will sign up for a voluntary endeavor that risks highlighting their weaknesses in public. But it certainly moves universities in the right direction.

Indeed, a strong case can be made that highlighting weaknesses could actually have a salutary effect on many American universities. When the OECD’s secondary school PISA test debuted in 2001, it showed that U.S. high school students were far behind many of their global counterparts. Finding out that America’s K–12 education was lagging behind the rest of the developed world didn’t hurt U.S. primary and secondary schools—it pushed them to make needed reforms. So far, U.S. colleges have little to fear from the currently available international rankings, which focus heavily on the research and reputation measures at which the long-established and top tier of American schools excel. But new rankings that shine a spotlight on student learning as well as research could deliver far less pleasant results, both for American universities and for others around the world that have never put much focus on classroom learning.

That doesn’t mean U.S. institutions should follow the advice of many in American higher education and try to steer clear of assessments such as AHELO. Such a move would only preserve U.S. schools’ international reputations in the short term; if the rest of the world cooperates with the OECD assessments, claims of American exceptionalism will look absurd. Furthermore, if the news AHELO brings about American higher education is worse than expected, the United States will be better off knowing it sooner rather than later. AHELO could be an instrument of much-needed change in the teaching side of American higher education, a useful way to get around the recalcitrance of those educational institutions that resist attempts at bringing some accountability to their multibillion-dollar enterprise. Even more broadly, as Merisotis argues, improved rankings can help universities innovate and thus maintain their standing in the face of intensified competition from upstart institutions. In undergraduate education, for instance, he suggests that a university that did away with the conventional credit system based on “seat time” in classrooms, moving instead toward measuring learning outcomes—the kind of content knowledge emphasized in the OECD’s nascent AHELO assessment system—could quickly establish itself as an entrepreneurial front-runner in the global brain race.

Ultimately, it is unrealistic to imagine that complaining about rankings and lamenting their popularity will do anything to slow their growth and appeal. Moreover, despite their shortcomings, their proliferation is a healthy indicator of a well-functioning—and burgeoning—global education marketplace. That universities so often oppose rankings reflects, says Kevin Carey of the think tank Education Sector, “an aversion to competition and accountability that ill serves students and the public at large.”⁷³ The real challenge will be how to improve rankings in order to give better guidance to students about university quality, and to provide more quality-driven incentives to universities that are eager to improve their national or international standing in the global academic race. If rankings ultimately spur universities to improve the quality of the research they produce, as well as the education their students receive, then the much-disparaged metrics will have succeeded, naysayers to the contrary, in providing their very own version of added educational value.