Part 2

Prospects for an Online Fix

HAVING PROVIDED WHAT I hope is a useful context, I will now discuss the prospects for using new technologies to address the productivity, cost, and affordability issues that I have described. I regard the prospects as promising, but also challenging. To succeed we will need to adopt a system-wide perspective, be relentless in seeking evidence about outcomes and costs, change some of our mindsets and our decision-making processes, and exhibit more patience than is our wont. None of these conditions is easy to satisfy! My focus will be on the contributions from established universities already serving large numbers of students. To be sure, we also want to serve new populations, at home and abroad, and a worldwide diffusion of knowledge is a most worthy goal—but it is not my central subject. Finally, in the search for new approaches, we need to recognize how well we do some things now, and how important it is that our educational institutions continue to stand for core values. That is the note on which I will end.

I am not a futurist but rather a maddeningly practical person who rarely has visions—and when I do, they are usually the result of having had a bad meal! But let me put such predilections to one side and ask readers to join me in imagining, just for a moment, how the intelligent harnessing of information technology through the medium of online learning might alter aspects of university life as we know it. Can we imagine a university in which

•   faculty collaborate more on teaching (with technology serving as the forcing function)?

•   faculty devote more of their time to promoting “active learning” by their students and are freed from much of the tedium of grading and even giving essentially the same lecture countless times?

•   students receive more, and more timely, individualized feedback on assignments?

•   instruction is guided by evidence drawn from massive amounts of data on how students learn, what mistakes students commonly make, and how misunderstandings underlying those mistakes can be corrected (“adaptive learning”)?

•   technology is used to bring the perspectives of a more diverse student body onto its campus through its capacity to engage students from around the world?

•   technology extends the educational process throughout one’s life through the educational equivalent of booster shots? And, ideally:

•   a university in which institutional costs and tuition charges rise at a slower rate?

Background

Before considering how nirvana might at least be approached (reaching nirvana will take a very long time, if indeed we can ever reach what has to be an ever-changing, ever-more ambitious goal), I want to describe briefly the evolution of my own thinking about technology and online learning. This story dates back at least as far as the Romanes Lecture that I gave at Oxford in 2000. In that lecture, I stressed the need to be realistic in thinking about how technology impacts costs, and I cited an early study from the University of Illinois that concluded: “Sound online instruction is likely to cost more than traditional instruction.”¹ I then cited a supporting observation from another early study: “A cyberprofessor trades the ‘chains’ of lecturing in a classroom for a predictable number of hours at a specific time and place for the more unpredictable ‘freedom’ of being accessible by email and other cyber technologies.… Many cybercourse instructors find themselves being drawn into an endless time drain.”² My conclusion at that time: “All the talk of using technology to ‘save money by increasing productivity’ has a hollow ring in the ears of the budget officer who has to pay for the salaries of a cadre of support staff, more and more equipment, and new software licenses—and who sees few offsetting savings.”³

I next added the not-so-profound thought that “this could change.” I am today a convert. I have come to believe that now is the time. Far greater access to the Internet, improvements in Internet speed, reductions in storage costs, the proliferation of increasingly sophisticated mobile devices, and other advances have combined with changing mindsets to suggest that online learning, in many of its manifestations, can lead to at least comparable learning outcomes relative to face-to-face instruction at lower cost. The phrase in many of its manifestations is important. Much confusion can result from failing to recognize that “online learning” is far from one thing—and that online learning is anything but static.⁴ It is, in fact, so many things and is evolving so rapidly that the efforts my colleague Kelly Lack and I made to create an understandable taxonomy did not succeed. We felt as if we were trying to “tether a broomstick,”⁵ and we decided to content ourselves with describing some distinguishing aspects of this complex landscape. (See the appendix at the end of this part.)

A far more sophisticated observer of digital trends than I am, President John Hennessy of Stanford, has been quoted as saying: “There’s a tsunami coming. [But] I can’t tell you exactly how it’s going to break.”⁶ Since I live on the East Coast, not the West Coast, I am even less capable of judging tsunamis, their shape, their force, or their timing, but I too am convinced that online learning could be truly transformative.

What needs to be done in order to translate could into will? The principal barriers to overcome can be grouped under three headings: the lack of hard evidence about both learning outcomes and potential cost savings; the lack of shared but customizable teaching and learning platforms (or tool kits); and the need for both new mindsets and fresh thinking about models of decision-making.⁷

The Lack of Hard Evidence

Prominent leaders in higher education have made it abundantly clear that the faculty and leadership of many institutions, especially those regarded as trend-setters, will consider major changes in how they teach if, and only if, much more hard evidence about potential gains is available. To be sure, better “facts” will not suffice to bring about change, but evidence may well be a necessary if not a sufficient condition.

How effective has online learning been in improving (or at least maintaining) learning outcomes achieved by various populations of students in various settings? Unfortunately, no one really knows the answer to either this question or the important follow-on query about cost savings. There have been literally thousands of studies of online learning, and my colleague, Kelly Lack, has continued to catalogue them and summarize their findings.⁸ This has proved to be a daunting task—and, it has to be said, a discouraging one. Very few of these studies are relevant to the teaching of undergraduates, and the few that are relevant almost always suffer from serious methodological deficiencies.⁹ The most common problems are small sample size; inability to control for ubiquitous selection effects; and, on the cost side, the lack of good estimates of likely cost savings in steady state.

Ms. Lack and I originally thought that full responsibility for this state of affairs rested with those who conducted the studies. We have revised that judgment. A significant share of responsibility rests with those who have created and used the online pedagogies, since the content often does not lend itself to rigorous assessment, and offerings are rarely designed with evaluation in mind. Moreover, the gold-standard methodology—randomized trials—is both expensive and excruciatingly difficult to implement on university campuses. Also at play is what I can only call the missionary spirit. The creators of many online courses are true believers who simply want to get on with their work, without being distracted by the need to do careful assessments of outcomes or costs. In all fairness, I have to add that these are early days, and it is unrealistic to expect to have in hand today careful assessments of potentially path-breaking offerings such as some of the MOOCs (massive open online courses) that have been introduced relatively recently.¹⁰ Still, there is no excuse for not working now on plans for rigorous third-party evaluations.¹¹

As Derek Bok has for years reminded everyone who will listen, the lack of careful studies of the learning effectiveness of various teaching methods is a long-standing problem; it predates and extends well beyond assessments of online learning. With typical candor, Professor William J. Baumol has observed: “In our teaching activity we proceed without really knowing what we are doing. I am teaching a course on innovative entrepreneurship, but I am doing so utterly without evidence as to the topics that should be emphasized, [or] the tools the students should learn to utilize. My state of mind on these matters is like that of 18th-century physicians, who used leeches and cupping to treat their patients simply because previous physicians had done so.”¹²

In an effort to fill part of this knowledge gap as it pertains to online learning, the ITHAKA organization mounted an empirical study of the learning outcomes associated with the use of a prototype statistics course developed by Carnegie Mellon University, taught in hybrid mode (with one face-to-face Q&A session a week).¹³ Carnegie Mellon’s course has several appealing features, including its use of cognitive tutors and feedback loops to guide students through instruction in basic concepts.¹⁴ In our study, we used a randomized trials approach, involving more than six hundred participants across six public university campuses, to compare the learning outcomes of students who took a hybrid-online version of this highly interactive course with the outcomes of students who took face-to-face counterpart courses. A rich array of data was collected at the State University of New York (SUNY), the City University of New York (CUNY), and the University System of Maryland. Although this study had limitations of its own, it is, we believe, the most rigorous assessment to date of the use of a sophisticated online course by the kinds of public universities that most desperately need to counteract the cost disease.¹⁵ I will cite only two principal findings about learning outcomes.

First, we found no statistically significant differences in standard measures of learning outcomes (pass or completion rates, scores on common final exam questions, and results of a national test of statistical literacy) between students in the traditional classes and students in the hybrid-online format classes (see figure 4).¹⁶ This finding, in and of itself, is not different from the results of many other studies. But it is important to emphasize that the relevant effect coefficients in this study have very small standard errors. One commentator, Michael S. McPherson, president of the Spencer Foundation, observed that what we have here are “quite precisely estimated zeros.” That is, if there had in fact been pronounced differences in outcomes between traditional-format and hybrid-format groups, it is highly likely that we would have found them.¹⁷

Figure 4 Effect of hybrid format on student learning outcomes

Note: None of the differences between traditional and hybrid courses was statistically significant at the 10 percent level.

Second, this finding is consistent not only across campuses, but also across subgroups of what was a very diverse student population. Half the students in our study came from families with incomes of less than $50,000, and half were first-generation college students. Fewer than half were white, and the group was about evenly divided between students with college GPAs above and below 3.0. The finding of consistent outcomes across this varied population rebuts the proposition that only exceptionally well-prepared, high-achieving students can succeed in online settings.¹⁸

Thus, while we did not find transformational improvements in learning outcomes, we did obtain compelling evidence that students with a wide range of characteristics learned just as much in the hybrid-online format as they would have had they instead taken the course in the traditional format.¹⁹ Students at the four-year universities in our study paid no price for taking a hybrid course in terms of pass rates or other learning outcomes. This seemingly bland result is, in fact, very important, in light of perhaps the most common reason given by faculty and deans for resisting the use of online instruction: “We worry that basic student learning outcomes will be hurt, and we won’t expose our students to this risk.” The ITHAKA research suggests that such worries may not be well founded—at least in situations akin to those we studied.

Several commentators on our study have expressed the hope—bordering on a conviction—that improvements in both the online offerings themselves and in the skill with which they are taught will lead, perhaps before long, to even better learning outcomes for such online offerings. The pace of change is dizzying, and improvements in pedagogy are almost certain to be made. In this regard, the pioneering work of the Khan Academy should be acknowledged. Salman Khan and his colleagues continue to learn a great deal about the most effective ways of using the online medium to teach.²⁰ More generally, institutions and their faculty are likely to become increasingly adept at using the feedback and other features of sophisticated online courses. Thus, there is, I agree, reason to believe that the findings we have reported for learning outcomes are a kind of baseline. Future studies may well obtain evidence that is even more favorable to continued experimentation with online approaches. But that is in the future and is simply another reason for advocating both continued development of these pedagogies and rigorous testing of their effectiveness.

What about cost savings? Whether pedagogies such as the one we tested can in fact reduce instructional costs, thereby lowering the denominator of the productivity ratio, is an absolutely central question—which is given even more prominence by our finding of equivalent learning outcomes at even this early stage in the development of sophisticated online systems. Because of its clear importance, we thought hard about how to estimate potential cost savings. But, truth be told, we did not do nearly as well in looking at the “cost” blade of the scissors as we did in looking at learning outcomes. We were able to do no more than suggest a method of approach and hazard what are little more than rough guesses (speculations) as to the conceivable magnitude of potential savings in staffing costs.

A fundamental problem, cutting across all types of online offerings, is that contemporaneous comparisons of the costs of traditional modes of teaching and of newly instituted online pedagogies are nearly useless in projecting steady-state savings—or, worse yet, highly misleading. The reason is that the costs of doing almost anything for the first time are very different from the costs of doing the same thing numerous times. That admonition is especially true in the case of online learning. There are substantial start-up costs associated with course development that have to be considered in the short run but that are likely to decrease over time, as well as transition costs involved in moving from the traditional, mostly face-to-face, model to a hybrid model of the kind that we studied. Instructors need to be trained to take full advantage of automated systems with feedback loops. Also, there may well be contractual limits on section size that were designed with the traditional model in mind but that do not make sense for a hybrid model. Such constraints have to be accepted in the short term, even though it may be possible to modify them over time.²¹

To overcome (avoid!) these problems, we carried out simulated cost probes. We conceptualized the research question not as “How much will institutions save right now by shifting to hybrid learning?” but rather as “Under what assumptions will cost savings be realized, over time, by shifting to a hybrid format, and how large are those savings likely to be?”

The crude models we employed (which ignore entirely the “joint products” issue that grows out of the practice of supporting graduate students as teaching assistants) suggest savings in compensation costs alone ranging from 36 percent to 57 percent when the traditional teaching mode relies on multiple sections.²² Of course, this simulation underestimates substantially the potential savings from moving toward a hybrid model because it does not account for space costs, which, in many instances, can dominate cost calculations. A fuller analysis would also deal with other infrastructure costs, some of which would undoubtedly be higher in a hybrid format, as well as take into account reductions in the time costs incurred by students.²³

Also highly relevant are the potentially profound effects of simplifications in scheduling and easier acceptance of transfer credits and evidence of prior learning. These could well lead, for many students, to an accelerated flow through the system, and thus to reduced time-to-degree and higher completion rates. If more students can be educated and if time-to-degree can be reduced, all without commensurate increases in costs, productivity could increase substantially via this avenue of impact.²⁴ Careful modeling of new scheduling possibilities, and of the implications for time-to-degree and completion rates, is definitely in order.

ITHAKA’s empirical study is undoubtedly helpful in overcoming skepticism about the learning outcomes of online offerings, but we must remember that it involved only one course, in a field well suited to online learning, in predominantly on-site contexts. We need many more careful studies of varied approaches to online learning, carried out in a variety of settings (including two-year colleges as well as a variety of four-year institutions). It is encouraging that the MOOC provider edX, with support from the Bill & Melinda Gates Foundation, is collaborating with two Boston-area community colleges to introduce an adapted version of an MIT online course in computer science (in what sounds like a hybrid mode).²⁵ It will be interesting to see how successfully this MIT offering can be adapted to the needs of quite different student populations. Careful assessment of learning outcomes, preferably by a third party, would be instructive.

Nor is it sufficient simply to compare outcomes of particular online offerings with outcomes in traditional face-to-face courses. We also need studies that compare the effectiveness of different approaches to online learning. We elected to test the Carnegie Mellon course because we believed that the highly interactive character of the CMU course, informed by cognitive science, was more promising than simpler approaches. But this course was expensive to develop.²⁶ Its value needs to be compared with the value of other approaches that are cheaper and less complex. It would also be highly desirable to compare outcomes and costs associated with various MOOCs against other approaches to online teaching. And we have to recognize that the answers to these questions about the costs and benefits of different approaches are likely to vary according to the content being presented, the student population, and the setting. ITHAKA, working with the MOOC provider Coursera and others, is engaged in just such a cross-platform study in collaboration with the University System of Maryland.²⁷

Designing research strategies in this area is a complicated business under the best of circumstances. Randomized trials are, it is generally agreed, the most promising way of reducing the ever-present risk of selection bias, but a huge takeaway from our empirical research using this methodology is that it is expensive and devilishly difficult to carry out. As we learned painfully, there are many important details that have to be worked out: how best to describe the course to be tested, how to recruit student participants in the study (including what incentives to use), how to randomize apprehensive students between “treatment” and “control” groups (and how to be sure that they stay in their assigned group), how to collect background information about student participants, and how to satisfy institutional review board (IRB) requirements in a timely way. Moreover, finding good answers requires the dayto-day involvement of campus staff not directly responsible to outsiders like us.²⁸ Looking ahead, I now think—heresy of heresies!—that the case for using randomized trials should itself be subject to careful cost-benefit analysis.²⁹ Appealing as they are, this may be an instance in which, in at least some cases, “the best is the enemy of the good.”³⁰

Last on my short list of research priorities is the evident need for creative analyses on the cost side of the ledger. This work should do more than just project direct costs (on a forward-looking, steady-state basis). It should include an analysis of implications for space utilization, capital costs, and indirect costs, hard as these are to estimate. It should also consider freshly the many ways in which online technologies may influence the way sizeable parts of the curriculum can be re-engineered (bearing in mind the injunction of the authors of the New England Journal of Medicine article about the need for such re-engineering, as in the much earlier introduction of electricity to manufacturing).³¹ The pace at which current students get through the educational system is enormously important, as are completion rates. It may be possible to utilize online technologies to allow energetic secondary school students to get an early start on their college education—perhaps by preparing them to take college-level tests that will allow them to place out of some introductory courses. Alternatively, given that the out-of-pocket costs of enrollment, and of attrition, are relatively low compared to the costs of traditional courses for which students need to pay tuition, MOOCs could allow students to experiment with different classes and get a sense of what disciplines interest them before they ever set foot on campus.³²

The biggest opportunity for MOOCs to raise productivity system-wide, and to lower costs, may well lie in finding effective ways for third parties to certify the “credit-worthiness” of their courses—and the success of students in passing them. The American Council on Education and Coursera have announced a pilot project that will entail faculty-led assessments of a subset of Coursera offerings. Students who complete courses that satisfy the requirements of the ACE’s standard College Credit Recommendation Service, and who pass an identity-verified exam, will be able to have a transcript submitted to the college of their choice. It will then be up to individual colleges to decide whether to give transfer credit for such work, though at present more than two thousand colleges and universities across the country generally accept ACE credit recommendations.³³ One key question to be addressed by colleges contemplating accepting such credentials will be how to define the coherence of the totality of the coursework that they expect students to complete in order to receive one of their degrees.

The Need for Customizable, Sustainable Platforms (or Tool Kits)

I now move on to discuss another type of need if we are to make real progress in utilizing technology in the pursuit of our aspirations. A major conclusion of ITHAKA’s Barriers to Adoption report is that “perhaps the largest obstacle to widespread adoption of ILO [interactive learning online]–style courses” is the lack at the present time of a “sustainable platform that allows interested faculty either to create a fully interactive, machine-guided learning environment, or to customize a course that has been created by someone else (and thus claim it as their own).” A companion conclusion is that “faculty are extremely reluctant to teach courses that they do not ‘own.’”³⁴ As one commentator cited in the Barriers report put it, “No one wants to give someone else’s speech” (even though all of us are happy to borrow felicitous phrases). This is by no means just about ego, although ego is certainly involved. Faculty may understandably feel that they are not sufficiently familiar with content prepared solely by someone else to teach it effectively. Also, both the structure of content and examples often need to be tailored to a particular student audience.³⁵

It would be easy—but incorrect—to infer from this line of argument, which emphasizes the need for customization, that the development of online courses has to be a responsibility of each individual campus. Sole reliance on purely homegrown approaches would be foolishly inefficient and simply would not work in most settings. It would not take advantage of the economies of scale offered by sophisticated software that incorporates features of well-developed platforms, including elaborate feedback loops and instructive peer-to-peer interactions.³⁶ Furthermore, many institutions simply do not have the money or the in-house talent to start from scratch in creating sophisticated online learning systems that can be disseminated widely. Nor would it make sense to re-invent wheels that can be readily shared.

There is clearly a system-wide need for sophisticated, customizable platforms (or tool kits) that can be made widely available, maintained, upgraded, and sustained in a cost-effective manner. Yet higher education thus far has failed to find a convincing solution to this problem, and immediate prospects for a solution are uncertain at best. In seeking to address this need, we must recognize the high probability that quite different pedagogies—and therefore somewhat different platforms—will be appropriate in subjects in which there are concrete concepts to be mastered and one right answer to many questions (for example, basic statistics)—as contrasted with discursive subjects which benefit from the exchange of different points of view (e.g., the Arab-Israeli conflict). At one point, I was much more inclined than I am at present to believe that a single platform or single tool kit might be appropriate. It now seems clear to me that the notion of a single dominant platform is unrealistic, given the entrepreneurial inclinations of numerous individuals and organizations. I now believe that such a notion is also unwise. There is much to be said for experimentation with different models and for competition among models as we search for approaches well suited to different needs. Adoption of any specific platform or platforms should be driven by a compelling strategy.

How should such platforms be developed? A strong prima facie case can be made for a high-level, collaborative effort within the traditional higher education community; after all, collaborations have been highly beneficial in sharing other assets, such as ultra-expensive scientific equipment. It is, however, widely recognized that collaborative efforts are difficult to organize, especially when much nimbleness is needed.³⁷ Collective decision-making is often cumbersome, and it can be hard to avoid lowest-common-denominator thinking. My favorite example is the Peace of Paris negotiations at the end of World War I, which ended so disastrously. John Maynard Keynes’s famous account of the collective efforts of the participants is worth recalling: “These then were the personalities of Paris—I forbear to mention other nations or lesser men: Clemenceau, aesthetically the noblest; the President, morally the most admirable; Lloyd George, intellectually the subtlest. Out of their disparities and weaknesses the Treaty was born, child of the least worthy attributes of each of its parents, without nobility, without morality, without intellect.”³⁸

There is, then, much to be said for seeking leadership from individual entities that are well-respected and have demonstrated a capacity to execute. Early on in our thinking about this issue, my colleagues and I were wondering whether Carnegie Mellon might address this need by scaling up the promising, highly interactive system that we tested and correcting the main shortcomings noted by participants in our study, including their interest in having a more customizable platform. Carnegie Mellon has expressed a commitment to developing the tools that are needed for authoring and analytics, which could well improve the scalability of their platform as we had originally hoped, but such an outcome is likely to be at least a year away.³⁹ In a field that is evolving as rapidly as this one, it remains to be seen how CMU’s cognitive-science, adaptive-learning approach will fit into the online learning landscape over the next few years.

We could, of course, simply let the marketplace provide: it is possible that for-profit entities, trading on financial incentives, might develop one or more effective platforms. There is, however, a risk that a for-profit might elect to cover some or all of its costs by essentially privatizing the significant amounts of information that such online systems can generate about how students learn. The example of Google illustrates dramatically the value that can be derived from exploiting a proprietary database for purposes such as selling targeted advertising. Massive amounts of data on how students learn can further the core mission of not-for-profit higher education and lead, in time, to the creation of better adaptive-learning systems. It would be unfortunate if the potential “public good” benefits of the rich information generated by online learning systems were lost. The educational community writ large should think hard about whether, and if so how, a depository for such information could be created and maintained. Such a depository should probably be created under the auspices of a nonprofit entity that recognizes explicitly the value of broad access to such critically important data. We should not think, however, that nonprofits are immune from the temptation to privatize and control data. There are too many examples, in publishing and in other fields, of situations in which nonprofits act very much like their for-profit cousins.

I have left for last what I regard as the most promising (though still entirely speculative) option at present: namely, the possibility that leading MOOCs might meet the need for readily adaptable platforms or tool kits. The developers of Coursera, edX, and Stanford’s Class2Go platform have said that they are committed to developing systems that can be used widely by others.⁴⁰ No one should doubt the good intentions of such entities. Nor should anyone undervalue the substantial resources at their disposal. It is precisely because they have a rare combination of assets—impressive technical capacity, a strong financial base, and real standing in the academic community (enhanced by extraordinary media coverage)—that I regard them, at least right now, as the highest-potential game in town. But neither should anyone underestimate the difficulty of modifying MOOCs originally designed to provide direct instruction to many thousands of individual students worldwide so that they can also serve the needs of existing educational institutions that serve defined student populations.⁴¹

The interviews we did for our Barriers study revealed “little enthusiasm for prepackaged online courses that did not permit customization, regardless of the institution ‘sponsoring’ the course, its quality, or the degree of interactivity.”⁴² And there is something of an inherent conflict, or at least a tension, between, on the one hand, the structure of MOOC offerings, which are designed largely by renowned and high-visibility professors at leading universities and which are generally provided worldwide on an “as-is” basis and, on the other hand, the need for at least some campus-specific customization. A related point is that the cost-effectiveness of MOOCs in their “direct to student” mode stems largely from the fact that their one-size-fits-all structure drives the marginal cost of serving even an extra thousand students close to zero. It is much less obvious how—or even whether—large cost savings can be achieved when a MOOC has to be customized for local use by a particular institution with a much smaller student population and a resident teaching staff. In addition, there are complex intellectual property rights issues that need to be resolved.⁴³

We should also recognize that while there has been much discussion about potential sources of revenue for MOOCs (charging for certificates of completion, becoming a kind of job-placement enterprise, and so on), the viability of the various hypothetical possibilities remains to be demonstrated.⁴⁴ A major lesson from the earlier MIT OpenCourseWare (OCW) experience is that it can be much easier to create something like OCW, often with philanthropic support, than to find regular sources of revenue to pay the ongoing costs of maintaining and upgrading the system. MIT is today still paying the running costs of OCW each year, and we are told that the faculty and trustees of MIT are convinced that they cannot go down the same path again—their pride in OCW as a truly pioneering venture notwithstanding.⁴⁵ Donor fatigue is a fact of life, and some regular, predictable source of revenue is needed for sustainability. There is real danger in announcing that something is free without knowing who is to pay the ongoing costs, which are all too real and cannot be ignored.⁴⁶ The “no free lunch” adage comes to mind.

These cautions and open questions about MOOCs cannot be ignored or assumed away. Nonetheless, I believe that the educational community should make every effort to take advantage of the great strengths of the leading MOOCs. Not only should we encourage their continuing interest in serving existing institutions as well as a global audience of individuals, but we should also try to find ways of testing learning outcomes and assessing cost-saving options for specific universities and university systems. It seems clear that MOOCs have an extraordinary capacity to improve access to educational materials from renowned instructors in various subjects for learners throughout the world. However, as far as I am aware, right now there is no compelling evidence as to how well MOOCs can produce good learning outcomes for 18- to 22-year-olds of various backgrounds studying on mainline campuses—and this is an enormous gap in our knowledge.

There is good reason to be extremely cautious in extrapolating crude findings for the student population that took the first MOOCs to mainline student populations. A highly preliminary study of the demographics of a MOOC in circuits and electronics, which at the time was offered by MITx but which is now offered by the larger edX organization, found that those students still around at the end of the course bore almost no resemblance to the students on mainline campuses in this country. One dramatic statistic: four out of five students who completed that MOOC had taken a comparable course at a traditional university prior to working their way through the MITx course. The circuits and electronics MOOC completers also differed along many other dimensions from traditional on-campus populations.⁴⁷

In addition to pursuing aggressively these key questions concerning learning outcomes on mainline campuses, the entire higher education community has an interest in thinking about business models that would assure the sustainability of the most promising MOOCs without compromising educational goals.⁴⁸ The experiences of entities such as JSTOR in developing sustainable business models could be relevant. Indeed, I suspect that at least part of the answer to the sustainability issue could lie in finding a JSTOR-like mechanism for charging reasonable fees to institutions (and/or students) that realize cost-saving benefits from MOOCs. Coursera and Antioch University have reached a content-licensing agreement that seems to move at least some distance in this direction.⁴⁹

The Need for New Mindsets—and Fresh Thinking about Decision-Making

My last category of challenges to be addressed is something of a grab bag—but a useful one, I hope. Many of the specific issues mentioned in the Barriers report share the attribute of requiring strong institutional leadership and even fresh ways of thinking about decision-making. These include, for example, the fact that “online instruction is alien to most faculty and calls into question the very reason that many pursued an academic career in the first place.… they had enjoyed being students, and valued the relationships that they enjoyed with their professors.”⁵⁰ Other barriers include the fear that online instruction will be used to diminish faculty ranks and the failure to provide the right incentives for faculty who are asked to lead online initiatives.

Hard as it sometimes is for beleaguered deans and presidents to confront challenges of these kinds directly, it is rarely wise to gloss over the most sensitive issues. I am convinced that a new, tougher, mindset is a prerequisite to progress. There is too strong a tendency to respond to financial pressures by economizing around the edges and putting off bigger—and harder—choices in the hope that the sun will shine tomorrow (even if the forecast is for rain!).

The seemingly unrelenting upward spiral of costs and tuition charges can be arrested, at least in some degree, only if presidents, provosts, and trustees make controlling both costs and tuition increases a priority. Academic leaders must look explicitly for strategies to lower costs. I am not saying that educational leaders lack courage (though, sadly, some do). The reality is that controlling costs is a hard sell, in part because strong forces are pushing in the opposite direction. As one of our advisers said, “Those opposed have so many ways of throwing sand in the wheels.”⁵¹

I continue to believe that the potential for online learning to help reduce costs without adversely affecting educational outcomes is very real. Absent strong leadership, however, there is a high probability that any productivity gains from online education will be used to gild the educational/research lily—as has been the norm for the last twenty years. Presidents and provosts should not mince words in charging their deans and faculty with teaching courses of comparable or superior quality with fewer resources—thereby either lowering the denominator of the productivity ratio or raising the student-learning component of the numerator (or both).⁵²

There is a definite political aspect to all of this. We must recognize that if higher education does not begin to slow the rate of increase in college costs, our nation’s higher education system will lose the public support on which it so heavily depends. There has been an undeniable erosion of public trust in the capacity of higher education to operate more efficiently.⁵³ In this respect, the better-off private and public universities—which rely heavily on many forms of federal support, including direct research grants from the National Institutes of Health, National Science Foundation, and other federal agencies; indirect cost recovery; financing of graduate students; and student loan guarantees—are in much the same boat as the more obviously endangered parts of the educational system. Efforts to save resources should be highly visible. Those who are skeptical about the capacity of established institutions to take positive steps in this sensitive area need to be given evidence that change is possible.

One favorable omen is the openness of many faculty to new ways of thinking—including the desirability of “flipping the classroom.”⁵⁴ A recent survey shows that a “decisive majority of professors”—69 percent—view with more excitement than fear the prospect of “changing the faculty role to spend less time lecturing and more time coaching students.”⁵⁵ Movement away from reliance on traditional lecturing, especially in large introductory courses, should allow institutions to devote the valuable in-person time of both faculty and students to activities that are more powerfully educational. Flipping the classroom need not save resources, however, and can even lead to higher costs. Lecturing is highly cost-efficient, whatever its educational shortcomings. Institutions that decide to flip the classroom should be well aware of this fact and should see if there are ways of saving resources by reducing the total amount of time spent in small group sessions (while, one would hope, improving the quality of the time that is spent). The hybrid version of the Carnegie Mellon statistics course that we tested is one possible model.

Growing openness to such concepts does not translate automatically, however, into new modes of teaching. Required is a willingness to question established norms, including models of decision-making. The challenges are at least as much conceptual, organizational, and administrative as they are technical. I wonder if the particular models of what is often called “shared governance” that have been developed over the last century are well suited to the digital world. Shared governance can mean dividing up tasks in seemingly clear-cut ways: leaving “corporate” decisions of one kind or another entirely in the hands of trustees and “academic” decisions entirely in the hands of faculty.⁵⁶ But if wise decisions are to be made in key areas, such as teaching methods, it is imperative that they be made by a mix of individuals from different parts of the institution—including faculty leaders but also others well-positioned to consider the full ramifications of the choices before them. There are real dangers in relying on the compartmentalized thinking that too often accompanies decentralized modes of organization to which we have become accustomed.⁵⁷

Given the institution-wide stakes associated with judgments as to when and how digital technologies should be used to teach some kinds of content, there is a strong case to be made for genuinely collaborative decision-making that includes faculty, of course, but that does not give full authority to determine teaching methods to particular professors or even to particular departments. There are too many spillover effects. It is by no means obvious that resources saved by using machine-guided learning in large introductory courses in subjects especially well suited to this approach should be captured in their entirety by the departments concerned.⁵⁸ It is important to think in terms of the institution as a whole in allocating savings—with prospective students and their parents among the stakeholders who deserve consideration. Also, the investments required to allow such savings—and to sustain initiatives—can be considerable and often have to be approved by a central authority.

Specific organizational solutions will vary from institution to institution, but the general principle is clear: some centralized calibration of both benefits and costs is essential. In a less complex age, it may have been sensible to leave almost all decisions concerning not just what to teach but how to teach in the hands of individual faculty members. It is by no means clear, however, that this model is the right one going forward, and it would be highly desirable if the academic community were seized of this issue and addressed it before “outsiders” dictate their own solutions. To repeat: faculty involvement is essential. There is a self-evident need for consultation with those who are expert in their disciplines and experienced in teaching—but this is not the same thing as giving faculty veto power over change.

Nor is this, I would emphasize, an issue of academic freedom, as that crucially important concept is properly understood. Faculty members should certainly be entirely free to speak their minds, as scholars and as teachers. But this freedom of expression should not imply unilateral control over methods of teaching. There is nothing in the basic documents explaining academic freedom to suggest that such control is included. It is not.⁵⁹ If academic freedom is construed to mean that faculty can do anything they choose, it becomes both meaningless and indefensible.⁶⁰

The case for genuinely fresh thinking about decision-making and shared governance rests only in part on the need for institutions to make carefully considered decisions about teaching methods in specific situations. It is also based on the fundamental nature of digital technologies, and the changes in the storage and dissemination of information, that are everywhere evident. One does not have to believe—as I certainly do not—that methods of online teaching, including MOOCs, will lead to diminished interest in high-quality residential education to recognize that the broader structure of higher education is likely to change quite profoundly. What we are witnessing is the early stage of at least a partial unbundling of activities that used to be the responsibility of a single faculty member or of groups of faculty in a single campus location: creating knowledge, teaching content, testing students’ grasp of that content, and credentialing.⁶¹ Joseph Aoun, president of Northeastern University, describes this as a “vertically integrated model,” and he contrasts it with the rise of “horizontal” models in which different people may perform some of these core functions—and perform them in different locations and for much larger numbers of students.⁶²

In the more horizontal model that Aoun and others envision, the roles of some faculty may be quite different than they are today. Different mixes of talents and inclinations will be required, and the organizational and administrative questions will be challenging. Some people and some institutions will undoubtedly resist such changes—some wisely, some unwisely. The New York University professor and writer Clay Shirky compares what is going on in some parts of education today with what happened to the recording industry when large numbers of people started to listen to MP3s. He describes this pattern as “a new story rearranging people’s sense of the possible, with the incumbents the last to know.… First, the people running the old system don’t notice the change. When they do, they assume it’s minor. Then that it’s a niche. Then a fad. And by the time they understand that the world has actually changed, they’ve squandered most of the time that they had to adapt.”⁶³

No one can say confidently how powerful or how pervasive the horizontal model will turn out to be—how many colleges and universities will be affected, and in what ways. But it seems clear that all of us interested in the future of colleges and universities, and in preserving key roles for faculty, are well advised to do our best to get some distance “ahead of the wave” (to return to John Hennessy’s tsunami metaphor).⁶⁴ It is an open question how the structure of decision-making and the definition of shared governance need to be modified in light of these changes brought on by the still rapidly evolving Internet age.⁶⁵ And it is a question that those of us in the academy, or with close ties to it, should be pondering now, before more time elapses.

What Must We Retain?

Let me now circle back to what I said earlier. As we contemplate a rapidly evolving world in which greater and greater use will surely be made of online modes of teaching, I am convinced that there are central aspects of life on our traditional campuses that must not only be retained but even strengthened. I will mention three.

First is the need to emphasize—and, if need be, to re-emphasize—the great value of “minds rubbing against minds.” We should resist efforts to overdo online instruction, important as it can be. There are, of course, both economic constraints and practical limitations on how much education can be delivered in person. But those of us who have benefited from personal interactions with brilliant teachers (some of whom became close friends), as I certainly have, can testify to the inspirational, life-changing aspects of such experiences. The half-life of content taught in a course can be short, as we all know; but great teachers change the way their students see the world (and themselves) long after the students have forgotten formulas, theorems, and even engaging illustrations of this or that proposition.⁶⁶ Moreover, a great advantage of residential institutions is that genuine learning occurs more or less continually, and as often, or more often, out of the classroom as in it. This cliché, repeated by countless presidents and deans, conveys real truth. Late-night peer-to-peer exchanges offer students hard-to-replicate access to the perspectives of other people. As one of my greatest teachers, Jacob Viner, never tired of warning his students, “There is no limit to the amount of nonsense you can think, if you think too long alone.”

My plea is for the adoption of a portfolio approach to curricular development that provides a carefully calibrated mix of instructional styles. This mix will vary by institutional type, and relatively wealthy liberal arts colleges and selective universities can be expected to offer more in-person teaching than can many less privileged institutions. However, even the wealthiest, most elite colleges and universities that seemingly can afford to stay pretty much as they are, at least in the short run, should ask if failing to participate in the evolution of online learning models is to their advantage, or even realistic, in the long run.⁶⁷ Their students, along with others of their generation, will expect to use digital resources—and to be trained in their use. And as technologies grow increasingly sophisticated, and we learn more about how students learn and what pedagogical methods work best in various fields, even top-tier institutions will stand to gain from the use of such technologies to improve student learning.

Second, we must retain, whatever the provocations, the unswerving commitment of great colleges and universities to freedom of thought—as exemplified so clearly by my great friend of so many years, Richard Lyman, Stanford’s seventh president, who died in May of this year. President Lyman stood resolutely for civility and protection of the rights of all. When he was compelled to summon the police to curb an over-the-edge demonstration in 1969, his action was applauded by some, but he thought the applause was misplaced. President Lyman said: “Anytime it becomes necessary for a university to summon the police, a defeat has taken place. The victory we seek at Stanford is not like a military victory; it is a victory of reason and the examined life over unreason and the tyranny of coercion.”⁶⁸

Third, our colleges and universities should focus unashamedly on values as well as on knowledge—and we should spend more time than we usually do considering how best to do this.⁶⁹ This is most definitely not a plea for pontificating. When President Robert Hutchins was urged to teach his students at Chicago to do this, that, or the other thing, he demurred, explaining: “All attempts to teach character directly will fail. They degenerate into vague exhortations to be good which leave the bored listener with a desire to commit outrages which would otherwise have never occurred to him.”⁷⁰

Let me now refer to a baccalaureate address given at Princeton in 2010 by Jeff Bezos, the CEO of Amazon, titled “We Are What We Choose.”⁷¹ Bezos began by reciting a poignant story of a trip he took with his grandparents when he was ten years old. While riding in their Airstream trailer, this precocious ten-year old laboriously calculated the damage to her health that his grandmother was doing by smoking. His conclusion was that, at two minutes per puff, she was taking nine years off her life. When he proudly told her of his finding, she burst into tears. His grandfather stopped the car and gently said to the young Bezos: “One day you’ll understand that it’s harder to be kind than clever.” Bezos went on in his address to talk about the difference between gifts and choices. “Cleverness,” he said, “is a gift, kindness is a choice. Gifts are easy—they’re given, after all. Choices can be hard.” Colleges and universities can, and should, find ways to help their students learn this key distinction—and encourage them, at least some of the time, to choose kindness over cleverness.

I return, finally (which one of my friends called the most beautiful word in the English language), to the question posed at the outset of this book: is online learning a fix for the cost disease? My answer: no, not by itself. But it can be part of an answer. It is certainly no panacea for this country’s deep-seated educational problems, which are rooted in social issues, fiscal dilemmas, and national priorities, as well as historical practices. In the case of a topic as active as online learning, we should expect inflated claims of spectacular successes—and of blatant failures. The findings I have reported warn strongly against too much hype. What Kin Hubbard famously said about those who claim certain knowledge of the currency question can be applied to online learning: “Only one fellow in ten thousand understands the currency question, and we meet him every day.” There is a real danger that the media frenzy associated with MOOCs will lead some colleges and universities (and, especially, business-oriented members of their boards) to embrace too tightly the MOOC approach before it is adequately tested and found to be both sustainable and capable of delivering good learning outcomes for all kinds of students.⁷²

Uncertainties notwithstanding, it is clear to me that online systems have great potential. Vigorous efforts should be made to explore further uses of both the relatively simple systems that are proliferating all around us, often to good effect, and more sophisticated systems that are still in their infancy—systems sure to improve over time, perhaps dramatically. In these explorations, I would urge us not to hesitate to experiment, but always to insist on assessments of outcomes. I would also urge us to think in terms of system-wide approaches and to exercise that rarest of virtues, patience. The careful development (and testing) of promising new pedagogies can take years and even decades.⁷³

I will end with a last story, on this theme of patience. It comes from the Arabian Nights, and I owe it to a very wise man, Ezra Zilkha, who was born in Baghdad. This is the story of the black horse. A prisoner who was about to be executed was having his last audience with the sultan. He implored the sultan: “If you will spare me for one year, I will teach your favorite black horse to talk.” The sultan agreed immediately with this request, and the prisoner was returned to his quarters. When his fellow prisoners heard what had happened, they mocked him: “How can you possibly teach a horse to talk? Absurd.” He replied: “Wait a minute. Think. A year is a long time. In a year, I could die naturally, the sultan could die, the horse could die, or, who knows, I might teach the black horse to talk.” The lesson of the story, Mr. Zilkha said, is, “If you don’t have an immediate answer, buy time. Time, if we use it, might make us adapt and maybe, who knows, find solutions.” If speaking to a college or university audience such as this one, Mr. Zilkha would add: “It is the job of the Stanfords of this world to teach the black horse to talk.”

 

APPENDIX: THE ONLINE LEARNING LANDSCAPE

Contours

At one end of this highly variegated landscape is an extremely large number of relatively straightforward online courses that provide an assortment of instructional materials on the web, often including videos, practice problems, and homework assignments. These courses (and some entire degree programs based on them) are usually institution-specific and built on learning management systems; they can be aimed at students in residence, distance learning populations, or both. They usually carry credit and are offered by both for-profit universities such as the University of Phoenix and a wide variety of nonprofit educational institutions. Some such courses in the nonprofit sector—not all of them entirely or even mostly online—have been created with the assistance of the National Center for Academic Transformation (NCAT) through its course-redesign initiative, which itself involves different models of online instruction.⁷⁴

According to a January 2013 report by the Babson Survey Research Group, the Sloan Consortium, and Pearson, about one in three college students now takes at least one online course (compared with about one in ten in fall 2002, the first year the survey was administered), and whereas total enrollments in higher education declined between fall 2010 and fall 2011, online enrollments grew about 9 percent during that time period.⁷⁵ Indeed, the current spread of online offerings is dizzying. During one week in August 2012, I came across announcements of online initiatives by the University of Florida system, a Seminole tribe program also in Florida (the Native Learning Center), University of Kansas, Utah Valley University, and a number of HBCUs whose activities were reported by the Digital Learning Lab of Howard University. (Websites are the best way to learn about these, and other initiatives too numerous even to mention here.) In addition, there are many online courses overseas, and the Open University in the United Kingdom has been especially active in this field for years.⁷⁶ In September 2012, Indiana University announced IU Online, a major new initiative that builds on a long history of work at that university and illustrates what is happening at a variety of institutions throughout the country.⁷⁷

The proliferation of offerings called online surely qualifies as a tidal wave, if not yet a tsunami. In addition to courses that can be counted, all of us feel the pervasiveness of the Internet in higher education by the increasing use of it in standard course-management systems, virtual reading materials, and a rapidly proliferating number of more and more sophisticated electronic textbooks incorporated into the curriculum. Even courses that are called “traditional” almost always involve some use of digital resources.

Carnegie Mellon University deserves special mention as a pioneer in the development of highly interactive online courses that have been built by teams of cognitive scientists, software engineers, and disciplinary specialists under the leadership of Candace Thille’s Open Learning Initiative. These OLI courses feature cognitive tutors—which draw on principles from cognitive psychology to guide a student through learning activities, taking into account the student’s progress—and three types of feedback loops: system to student, providing instant feedback to students on their answers to problems and carefully structured hints as to how to get the right answers; system to teacher, providing current information to the teacher on how individual students, as well as students in general, are doing (thereby enabling teachers to make more effective use of any face-to-face time that is available); and system to course designer, providing information on parts of the course that are working well and parts that need improvement.⁷⁸

At still another corner of this landscape are the MOOCs (massive open online courses)—usually designed by highly regarded professionals and taught to thousands of students worldwide by well-known professors. Typically, students who have registered for these courses (usually for free) watch videos and complete assignments that are machine-graded or graded by other students and/or teaching assistants. With very few exceptions, these courses do not carry college credit or lead to degrees, and they may or may not lead to certificates of accomplishment or badges (for which students may need to pay a modest fee) that indicate mastery of particular skills. Three of the best-known exemplars of MOOCs are listed below:

•   Coursera, a for-profit spin-off from Stanford that offers a wide variety of courses in close collaboration with several dozen high-profile universities (including Princeton, the University of Toronto, and the University of Michigan, as well as Stanford), to which Coursera provides authoring tools and other forms of assistance;

•   Udacity, another for-profit Stanford spin-off, which concentrates on computer science and related fields; unlike Coursera, Udacity works only with individual professors (rather than through institutions);

•   edX, a nonprofit partnership of MIT, Harvard, the University of California at Berkeley, Georgetown, Wellesley, and the University of Texas System that offers courses of its own, initially focusing mainly on computer science and engineering, and which also plans to make its platform available on an open-source basis to faculty elsewhere who wish to create their own courses.⁷⁹

Again, websites are the best source of information about these and other MOOCs.

Another well-known provider of free online course materials is Khan Academy, a nonprofit organization which is perhaps best known for its short instructional videos hosted on YouTube, but which today emphasizes automated practice exercises that are used heavily by secondary school students. Its instructional videos cover a broad range of disciplines, ranging from civics and art history to computer science, chemistry, differential equations, and the Greek debt crisis, though it has generally been Khan’s mathematics materials that are used in classroom settings. While one can argue about whether Khan Academy should be classified as a MOOC, in light of the fact that its typical offerings are not courses, the breadth and widespread appeal of the Khan Academy’s offerings undoubtedly bear mention.

In mid-November 2012, a consortium of ten prominent universities announced that it will offer credit-bearing online courses, starting in fall 2013, as part of an initiative called Semester Online.⁸⁰ The consortium will operate through the platform developed by the educational technology company 2U. Both students enrolled in the member institutions and students outside those institutions will be able to take the thirty or so courses offered by the consortium members for credit, though students outside the host universities will have to apply to take the courses and, if admitted, pay tuition for those courses. Unlike MOOC providers, this consortium will emphasize small, live “virtual” discussion groups in which students interact with each other and with the instructor in real time. The small class sizes and synchronous nature of the interactions are likely to make these offerings more expensive than MOOCs and some other online courses.

Distinctions

In contemplating the wide array of offerings that populate the online universe, it may be helpful to think in terms of eleven overlapping distinctions, grouped under four headings.

Particular Features of Online Courses

  1.  How advanced is the content of the course, and are there daunting prerequisites?

  2.  To what extent does the course contain cognitive tutors (akin to those available in Carnegie Mellon’s OLI courses) or other adaptive learning features?

  3.  To what extent does the course allow learners to interact with each other, and perhaps with instructors or teaching assistants who are leading or overseeing the course (if there are any)?

How Content Is Delivered

  4.  Is the course purely online, or is it a hybrid course with a face-to-face element?

  5.  Is the online component of the course offered in synchronous mode (that is, do students all have to be online at the same, specified times), or in asynchronous mode (where students can access the materials any time they choose), or both?

Entities Offering the Courses and the Intended Audiences

  6.  Is the course offered “direct to student” or through an existing college or university?

  7.  What is the primary intended student population—working adults in the United States (who more often than not study part-time), more traditional students (often but not always campus-based), or anyone and everyone with aptitude and interest all over the world?

  8.  To what extent can the course be adapted, or re-purposed, to serve other sets of students in the future, in various settings? There are important distinctions among online courses that are “home-grown,” designed on an institution-specific platform that has little customization capacity, and intended specifically for use by a known institutional population; online courses that are developed for a broader (and unknown) population of students; and online courses that are developed alongside, or on top of, a general platform but that have customizable features and allow for “local” varieties targeted at particular populations.

Credentialing and Ownership

  9.  Does the course offer credit and a path to a degree, a certificate of accomplishment, or no credit or assessment of accomplishment?

10.  Who owns (and/or has license to use) the intellectual property of the course materials? Is the controlling entity a for-profit or nonprofit organization?

11.  What is the business model underlying the course offering? Is the course available to students for free, and if it is, who pays for the development and ongoing operation of the course?

There are obviously hundreds of possible permutations and combinations involving these and other distinctions. With so many dimensions along which online courses can be classified, a simple taxonomy can be both elusive and more confusing than helpful. The variety of online offerings is often under-appreciated, as is the importance of deciding what characteristics are appropriate in a particular setting.

Notes

1. William G. Bowen, At a Slight Angle to the Universe: The University in a Digitized, Commercialized Age, Romanes Lecture for 2000, University of Oxford, October 17, 2000 (Princeton, NJ: Princeton University Press, 2001), 24 (also available online at www.mellon.org/internet/news_publications/publications/romanes.pdf); “Teaching at an Internet Distance,” University of Illinois faculty seminar, December 7, 1999, www.vpaa.uillinois.edu/tid/report.

2. Cited in Bowen, At a Slight Angle to the Universe, 23–24; Peter Navarro, “Economics in the Cyberclassroom,” Journal of Economic Perspectives 14, no. 2 (Spring 2000): 129.

3. See Bowen, At a Slight Angle to the Universe, 24.

4. In the words of the authors of one study comparing face-to-face instruction with three different varieties of distance learning: “Like Campbell’s Soups, distance learning now comes in so many varieties that it is increasingly difficult to generalize about it.” (James V. Koch and Alice McAdory, “Still No Significant Difference? The Impact of Distance Learning on Student Success in Undergraduate Managerial Economics,” Journal of Economics and Finance Education 11, no. 1 (Summer 2012): 36.)

5. This imagery is from Keynes’s explanation of his difficulty rendering a portrait of Lloyd George at the Peace of Paris. John Maynard Keynes, Essays in Biography (New York: Horizon Press, 1951), 33. I shall offer one other snippet from this remarkable essay when I discuss collective decision-making.

6. “Changing the Economics of Education,” interview with John Hennessy and Salman Khan, Wall Street Journal, June 4, 2012. See also Ken Auletta, “Get Rich U,” New Yorker, April 30, 2012; Billy Gallagher, “Q&A: President John Hennessy on Online Education,” Stanford Daily, October 30, 2012.

7. For a fuller study of barriers to adoption of online pedagogies, see Lawrence S. Bacow, William G. Bowen, Kevin M. Guthrie, Kelly A. Lack, and Matthew P. Long, Barriers to Adoption of Online Learning Systems in U.S. Higher Education, May 1, 2012, available on the ITHAKA website at www.sr.ithaka.org. The discussion in this book draws heavily on this report but is more cryptic and organizes the issues differently.

8. Kelly A. Lack, “Current Status of Research on Online Learning in Postsecondary Education.” This is a revised version of a paper originally dated May 18, 2012, available at www.sr.ithaka.org/research-publications/current-status-research-online-learning-postsecondary-education.

9. In the widely cited SRI/DOE meta-analysis (usually cited as Means et al., 2009), most of the forty-six studies reviewed involved online learning in the fields of medicine or health care, and a great many studies compared the use of the two different modes of learning for less than a full semester. In addition, only twenty-five of the fifty-one online versus face-to-face comparisons analyzed in the meta-analysis involved undergraduate students. (The other twenty-six involved students in grades K–12, graduate students, or other types of learners.) See Barbara Means et al., Evaluation of Evidence-Based Practices in Online Learning: A Meta-analysis and Review of Online Learning Studies (Washington, DC: U.S. Department of Education, 2009), www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.

10. In general, MOOCs are free or low-cost online courses that are available to interested users—in some cases, by the thousands, tens of thousands, or hundreds of thousands—throughout the world. These courses typically consist of video lectures by well-known professors or experts in a particular field, often affiliated with elite institutions; the video lectures generally are complemented by problem sets and/or other assignments, and, in some cases, discussion boards where students can interact with one another asynchronously. Students typically have very little opportunity to interact with professors (with the exception, in some cases, of mass e-mails sent by the instructor to all enrolled students), though some instructors have teaching assistants available to answer questions or monitor the discussion boards. Completion of a MOOC is sometimes recognized with a certificate of accomplishment from the professor or from the MOOC itself, though it generally is not attached to credit from the college or university with which the professor is affiliated. See Lack, “Current Status of Research on Online Learning,” for a description of varieties of online learning, including some of the best-known MOOCs. For an excellent summary of this terrain, see Laura Pappano, “The Year of the MOOC,” New York Times, November 4, 2012. For a slightly earlier overview of the characteristics of, and the recent developments among, some MOOCs, see Abby Clobridge, “MOOCs and the Changing Face of Higher Education,” Information Today, August 30, 2012, http://news​breaks​.info​today.com/News​Breaks/MOOCs-and-the-Changing-Face-of-Higher-Education-84681.asp. This field is changing so rapidly that there is one or another new development almost every day.

11. Khan Academy is undergoing an assessment, conducted by SRI International’s Center for Technology in Learning, regarding the adoption and effectiveness of its materials in classrooms at twenty-one primary and secondary schools in Northern California during the 2011–12 year; a June 2012 newsletter said that a report was expected in December 2012, though as of early January 2013 no report had been published. (See “Research Update,” SRI International, Center for Technology in Learning, June 2012, http://ctl​.sri​.com​/news​/news​letter​_june​_2012​/june​_2012_news.html.) With respect to the MOOCs offering college-level courses, both Coursera and edX have expressed an interest in working with ITHAKA on assessments, and a recent piece in the Chronicle of Higher Education reported that edX is planning to test a “flipped classroom” model—combining the use of content from its online courses with face-to-face teaching—at a community college. (Marc Parry, “5 Ways That edX Could Change Education,” Chronicle of Higher Education, October 1, 2012.) According to an e-mail from Harvard’s provost, Harvard has also developed “a course support model to assist faculty in thinking about what their own HarvardX courses might look like and to facilitate experimentation in course design” and is in the process of creating “a micro development environment” that will enable faculty to design courses and smaller pieces of courses called modules (Provost Alan M. Garber, e-mail to colleagues, November 16, 2012). (HarvardX is the Harvard organization responsible for the university’s edX contributions and activities.) For-profit publishers active in this field have assembled some results that they claim, not surprisingly, make a case for their products. Disinterested third-party assessments are clearly in order. See the appendix material in Lack, “Current Status of Research on Online Learning.”

12. William J. Baumol, personal communication, November 7, 2012.

13. ITHAKA is a nonprofit organization created initially by the Andrew W. Mellon, William and Flora Hewlett, and Stavros Niarchos foundations. It is the parent of JSTOR and Portico and also operates an increasingly important strategy and research division (Ithaka S+R). Kevin M. Guthrie is the president of ITHAKA, and its board is chaired by Henry S. Bienen, president emeritus of Northwestern University. ITHAKA’s mission is “to help the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways.”

14. The director and vice provost of the Carnegie Mellon’s Open Learning Initiative (OLI) courses define a “cognitive tutor” as “a computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a human tutor, that is, making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.” They further explain that, unlike “traditional computer aided instruction,” which “gives didactic feedback to students on their final answers,” cognitive tutors “provide context-specific assistance during the problem-solving process.” (Candace Thille and Joel Smith, “Learning Unbound: Disrupting the Baumol/Bowen Effect in Higher Education,” Futures Forum, American Council on Education 2010, http://oli​.cmu​.edu​/wp-oli/wp-content/up​loads​/2012/05/Thille​_2010​_Learning_Unbound.pdf, 34).

The director and vice provost also outline the process by which “feedback loops” can be used to support classroom learning: an instructor first assigns students to work through part of the OLI course, during which time the OLI system gathers, analyzes, and organizes data about the students’ activities; the OLI system then presents the instructor with information about her students’ learning for the instructor to review; and the instructor adapts her teaching as needed (ibid., 36).

15. See William G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren, “Interactive Learning Online at Public Universities: Evidence from Randomized Trials,” May 22, 2012, available on the ITHAKA website at www.sr.ithaka.org. We are pleased to report that the study has been very well received by media outlets such as the National Review, the Boston Globe, the Chronicle of Higher Education, Inside Higher Ed, and, in particular, the Wall Street Journal (whose writer David Wessel called the report “carefully crafted” and its findings “statistically sound”). In addition to the six four-year institutions included in the study, we tried to include three community colleges. But for a variety of reasons—many logistical—this effort did not succeed, and we caution readers against simply extrapolating our findings to two-year colleges.

Courses like the CMU statistics course tested in this study exemplify what we call the interactive learning online (ILO) approach. This approach contrasts with more common types of online learning which often mimic classroom teaching without taking advantage of the unique online environment to provide “added value.”

16. As can be seen from the figure, hybrid-format students did perform slightly better than traditional-format students on three outcomes—achieving pass rates that were about three percentage points higher, scores on the Comprehensive Assessment of Outcomes in Statistics (CAOS) that are about one percentage point higher, and final exam scores that are two percentage points higher—but none of these differences passes the usual tests of statistical significance. (The CAOS test is a standardized, 40-item multiple-choice assessment designed to measure students’ statistical literacy and reasoning skills. In our study, we administered the CAOS test once at the beginning of the semester and again at the end of the semester. One characteristic of the CAOS test is that, for a variety of reasons, scores do not increase by a large amount over the course of the semester; among all the students in our study who took the CAOS test at both the beginning and the end of the semester, the average score increase was five percentage points. For more information about the CAOS test, see http://app.gen.umn.edu/artist/caos.html and Robert delMas, Joan Garfield, Ann Ooms, and Beth Chance, “Assessing Students’ Conceptual Understanding after a First Course in Statistics,” Statistics Education Research Journal 6, no. 2 (2007): 28–58.)

17. Thus, our finding is strikingly different in this consequential respect from an alternative (hypothetical) finding of “no significant difference” resulting from a coefficient of some magnitude accompanied by a very large standard error. A finding with big standard errors would mean, in effect, that we just don’t know much—the “true” results could be almost anywhere.

18. We wondered if the opposite proposition would hold—that is, we thought it possible that students who are subject to what Claude Steele has called stereotype threat might actually do better in more anonymous settings. Not proven, is the verdict of this study. The size of our study, with over six hundred participants—roughly half in the “treatment” sections and half in the “control” sections—allowed us to look more carefully than most other studies have been able to do at these more refined groupings of students. We calculated results separately for subgroups of students defined in terms of characteristics including race/ethnicity, gender, parental education, primary language spoken, pretest score (that is, score on the first administration of the CAOS test), hours worked for pay, and college GPA. We did not find any consistent evidence that the hybrid-format effect varied by any of these characteristics (see Bowen et al., “Interactive Learning Online at Public Universities,” table A6).

19. We also found, however, that students had a mild preference for traditional face-to-face instruction and that those in the hybrid-format sections thought that they had learned less, even though objective outcomes were essentially the same for students in the two groups. (See Bowen et al., “Interactive Learning Online at Public Universities,” figure 3 and table A7.) A leader of one of the universities that participated in our study opined that a defect of the CMU prototype course is that it has no “addictive” or “Disney-like” appeal; it was, as this person put it, “designed by cognitive scientists” (no offense intended!). In contrast, some students in the traditional format may have been treated to an occasional colorful story, personal recollections of the instructor, or other practices that are sometimes used by faculty to make class more enjoyable and that have the effect of improving students’ opinions of their course. The question of what is really going on here—with no differences in learning outcomes, as measured conventionally, combined with a (to be sure, small) difference in qualitative assessments—relates to a larger literature on objective measures of learning outcomes versus more subjective measures of student satisfaction (citations given in Bowen et al., “Interactive Learning Online at Public Universities,” and in Lack, “Current Status of Research on Online Learning in Post secondary Education”).

20. The Khan Academy is a nonprofit that offers a free online library of more than three thousand short video lectures on a variety of topics. For some subjects, particularly those related to mathematics, Khan also provides practice exercises and data tools for instructors. For more information about the Khan Academy, see the appendix to part 2 of this book and the extended discussion of Khan’s work in Michael Noer, “Reeducating Education,” Forbes, November 19, 2012. In addition to the Khan Academy, Coursera, edX, Udacity, and other MOOC providers are working hard to garner evidence about what teaching methods work best. The potential for progress is considerable.

21. The existence of these and other problems probably explains, but only in part, the surprising lack of attention to costs among those who have studied online learning. Unfortunately, proponents of online learning often seem uninterested in costs. Carol Twigg at the National Center for Academic Transformation (NCAT) is an exception to this statement, and she deserves great credit for having focused on costs earlier than most people. However, the NCAT cost studies have been almost entirely self-directed by the institutions involved, which is far from ideal. Also, it is unclear in many cases whether initial successes with these courses were sustained. (See the discussion of the limitations of the NCAT studies in Lack, “Current Status of Research on Online Learning.”)

22. The appendix material in Bowen et al., “Interactive Learning Online at Public Universities,” presents these results and many more calculations, along with some graphs showing how sensitive potential savings are as we vary assumptions about section sizes and compensation.

23. Our results indicate that hybrid-format students took about one-quarter less time to achieve essentially the same learning outcomes as traditional-format students (Bowen et al., “Interactive Learning Online at Public Universities,” 23).

24. Potential cost savings for both students and institutions are illustrated in an article on developments at the College of St. Scholastica, a parochial private college in Minnesota. See Paul Fain, “Another Push on Prior Learning,” Inside Higher Ed, November 19, 2012. This college is a member of the Council of Independent Colleges, and Richard Ekman, the Council’s president, has reported that several other member institutions are thinking along similar lines. This is a useful reminder that the private sector of higher education, including the small-college sector, has a big stake in these developments.

25. See John Lauerman, “Gates Backs MIT Online Course Offering at Community Colleges,” Bloomberg, November 18, 2012, www.bloomberg.com/news/2012–11–18/gates-backs-mit-online-course-offering-at-community-colleges.html.

26. For an instructive account of the history of this course, which enjoyed large-scale support from the William and Flora Hewlett Foundation, see Taylor Walsh, Unlocking the Gates (Princeton, NJ: Princeton University Press, 2011), chapter 4.

27. See “Informing Innovation in Higher Education: Evidence from Implementing the Latest Online Learning Technologies in a Public University System,” Ithaka S+R, November 13, 2012, www.sr.ithaka.org/news/informing-innovation-higher-education-evidence-implementing-latest-online-learning-technologies.

28. Bowen et al., “Interactive Learning Online at Public Universities,” contains a detailed discussion of the many lessons we learned along the way—including the importance of running pilots on each campus before conducting the research phase of the study. Others embarking on similar projects may find it valuable to ponder our missteps, most of which, fortunately, we were able to correct, following the pilots. We have great respect for other investigators who have coped with such problems, often in settings outside higher education.

29. Some careful work of this kind has been done. For example, an analysis by Shadish, Clark, and Steiner showed that, in some cases, the results of non-experimental studies can approximate those of experimental studies, particularly when a rich array of well-measured and well-chosen covariates is available, and when ordinary linear regression and/or propensity scoring are used to reduce bias. (See William R. Shadish, M. H. Clark, and Peter M. Steiner, “Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment,” Journal of the American Statistical Association 103 [2008] : 1334–56.) Similarly, when Shadish, Cook, and Wong (2008) examined twelve within-study comparisons of randomized and nonrandomized studies, they found that eight of these comparisons produced “reasonably close” results—with two of the remaining four comparisons having close results with respect to some analyses but not others, and the final two comparisons involving “particularly weak observational stud[ies].” (Thomas D. Cook, William R. Shadish, and Vivian C. Wong, “Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons,” Journal of Policy Analysis and Management 27, no. 4 [2008]: 724–50.) More specifically, Shadish and his colleagues found that the three studies involving a regression-discontinuity design “produced essentially the same statistical significance patterns” as long as their analyses used the same assumptions as the experimental designs; that they “[could] also trust” the results from observational studies that minimized initial differences by matching intact control and treatment groups using some sort of baseline measure; and that even in cases where it was not possible to match treatment and control groups, identifying and measuring the correct selection process through the use of ordinary least squares regression, instrumental variables, or propensity score analyses allowed for the reduction of selection bias to some extent. The researchers concluded, “Taken as a whole, then, the strong but still imperfect correspondence in causal findings reported here contradicts the monolithic pessimism emerging from past reviews of the within-study comparison literature.”

30. I am certainly not suggesting, however, that we abandon the search for rigor. I am suggesting that careful consideration be given to finding simpler approaches (as suggested in the previous note) that approximate the randomized trials model—perhaps by the use of well-chosen matching methods or lotteries in situations in which face-to-face courses are oversubscribed. There is also much to be said for quasi-experimental studies that use cut-offs and regression-discontinuity approaches. As my colleague Kevin M. Guthrie keeps pointing out, any kind of side-by-side test of different teaching methods is beset by complications. Ironically, it is much less problematic (though less instructive) just to substitute an entirely new approach for what was there before. Manifold issues that concern institutional review boards are thereby avoided. This is a bizarre state of affairs that deserves examination.

31. I refer here to the same NEJM article cited in part 1: Spencer S. Jones, Paul S. Heaton, Robert S. Rudin, and Eric C. Schneider, “Unraveling the IT Productivity Paradox—Lessons for Health Care,” New England Journal of Medicine 366 (June 14, 2012): 2243–45.

32. See Daphne Koller and Andrew Ng, “Taking the Next Step in Online Education with Credit Equivalency,” Forbes, November 16, 2012. At the same time, the time costs associated with taking MOOCs, which often require ten or more hours of work per week, are not negligible, and there are many students, particularly those who have part- or full-time jobs, who may simply not have the time to spend exploring new subjects through MOOCs, especially if they cannot earn credit for MOOC completion.

33. See Tamar Lewin, “College Credit Eyed for Online Courses,” New York Times, November 13, 2012, as well as the official announcement of this project by Coursera and the discussion of this collaboration in Koller and Ng, “Taking the Next Step.” It is encouraging that a number of MOOCs are moving to address worries about cheating by requiring students to submit pictures of themselves with a form of photo identification via webcam, by sampling and comparing students’ keystroke patterns, by arranging for either remote proctoring services or on-site proctoring of exams, or by other means. This development will surely increase the odds that some educational institutions will give credit, or at least advanced standing, to students who earn “certificates” of accomplishment or “badges” of some kind. (See Tamar Lewin, “Colorado State to Offer Credits for Online Class,” New York Times, September 7, 2012; Steve Kolowich, “Site-Based Testing Deals Strengthen Case for Granting Credit to MOOC Students,” Inside Higher Ed, September 7, 2012; Ki Mae Heussner, “5 Ways Online Education Can Keep Its Students Honest,” GigaOM, November 17, 2012, http://gigaom.com/2012/11/17/5-ways-online-education-can-keep-its-students-honest/; and Nick Anderson, “Online College Courses to Grant Credentials, for a Fee,” Washington Post, January 9, 2013.)

Kevin Carey of the New America Foundation has also cited the strong possibility that, over time, MOOCs will gain more and more acceptance, which he believes will lead to some “disintermediation” of educational services (separating credentialing from teaching) and, in turn, some cost savings for students and perhaps institutions. (Kevin Carey, “Into the Future with MOOCs,” Chronicle of Higher Education, September 3, 2012.)

34. See Bacow et al., Barriers to Adoption, 21.

35. For a fascinating account of how one obviously talented professor of computer science at Vanderbilt University navigated this terrain, see Douglas H. Fisher, “Warming up to MOOCs,” Chronicle of Higher Education, Professor Hacker blog, November 6, 2012. After describing his own initial reluctance to use the content of MOOCs created by others in his own courses (using much the same language that we used in our Barriers to Adoption report), Fisher decided that this attitude was just “silly,” and he goes on to describe in great detail how he used Stanford course content highly effectively to “flip” his own classes. But Fisher also created his own “wrappers” for the material and obviously customized the content. He writes eloquently about how exciting he has found embracing the online content of others after years of being “alone in the wilderness.” He now feels, he reports, as if he is part of a rich scholarly community whose members build on one another’s best work. This is a highly praiseworthy attitude, I would submit, but it also reflects the considerable accomplishments and self-confidence of Professor Fisher—qualities not likely to be shared by anything like all professors at other colleges and universities.

36. Stanford president John Hennessy has lauded the social networking aspects of MOOCs as a source of added value, relative to what may be gained from more solitary online courses (as well as some face-to-face courses), specifically calling the speed with which MOOC students responded to each others’ discussion board posts “phenomenal.” (See Salman Khan and John Hennessy, “Changing the Economics of Education,” Wall Street Journal, June 4, 2012.)

Princeton professor Mitchell Duneier has described his experience teaching a Coursera course in similarly enthusiastic terms. (Mitchell Duneier, “Teaching to the World from Central New Jersey,” Chronicle of Higher Education, September 3, 2012; see also quotations from Duneier in Tamar Lewin, “College of the Future Could Be Come One, Come All,” New York Times, November 19, 2012.) In the New York Times article, Duneier is quoted as saying that within three weeks of starting to teach his course in introductory sociology, “I had more feedback on my sociological ideas than I’d had in my whole teaching career. I found that there’s no topic so sensitive that it can’t be discussed, civilly, in an international community.”

EdX’s Anant Agarwal also offers a helpful account of the value of peer-to-peer responses to questions in an online setting, describing the “fascinating” speed with which MOOC students answer each others’ questions—even at 2 A.M. (See Tamar Lewin, “One Course, 150,000 Students: Q&A with Anant Agarwal,” New York Times, July 18, 2012.) An irony not to be missed is that from this point of view, the more students, the better—in contrast to the usual desire to reduce class size in traditional teaching.

37. In his contribution to a Windsor Group study in 2007, President Hennessy both stressed the appeal of the idea of collaboration and explained why it is so hard to achieve when institutions have different needs and wish to exploit distinctive differences. (John Hennessy, “Technology and Collaboration: Creating and Supporting Public Goods,” draft memos from Windsor Working Groups, September 24, 2007.)

38. Keynes, Essays in Biography, 39. I have often thought that private, family-run foundations yield other examples of the perils of collective decision-making absent a clear sense of direction and some precise location of authority.

39. Personal communication from the very able leader of the CMU project, Candace Thille, to Ira Fuchs, a member of the ITHAKA board, September 25, 2012.

40. For instance, one of the models proposed for Coursera involves providing a version of its platform and course content to community colleges to use in for-credit, low-cost courses for their own students; another proposed model would involve students at a particular university taking proctored exams upon completion of Coursera courses, in order to verify their skills in a certain area (so that, for instance, these students could receive a course waiver).

Likewise, at the time it was founded as a joint initiative of MIT and Harvard, edX’s website said that edX would “begin by hosting MITx and Harvardx content, with the goal of adding content from other universities interested in joining the platform.” (The precursor of edX, MITx, featured only MIT course content and was launched about half a year before the announcement of edX. The names MITx and HarvardX are currently used to refer to the course content from MIT and Harvard, respectively, that is hosted on the edX platform.) Since the announcement of edX in May 2012, edX has expanded to include several additional institutions. (See the appendix for more information about edX and its partner institutions.) Among edX’s listed goals are to “expand access to education, allow for certificates of mastery to be earned by able learners, and make the open-source platform available to other institutions.”

Finally, the website of Class2Go, Stanford Online’s new internal platform, says that its creators “believe strongly that valuable course content shouldn’t be tied to any one platform” and that Class2Go is open source in order to encourage others to use it, or “to work together with similar efforts in other places.” (See Jeffrey R. Young, “Inside the Coursera Contract, How an Upstart Company Might Profit from Free Courses,” Chronicle of Higher Education, July 19, 2012; “Online Course Hosting and Servicing Agreement,” quoted in “The U. of Michigan’s Contract with Coursera,” Chronicle of Higher Education, July 19, 2012; “What is edX? Answering Common Questions about MIT and Harvard’s New Partnership in Online Education,” MIT News, May 2, 2012; and “Class2Go: Take Stanford Online Classes, Anywhere,” Stanford University, 2012, http://class2go.stanford.edu.)

41. I do not regard the question of whether such an entity is for-profit or nonprofit as anything close to dispositive in thinking about possibilities for meeting the need I have just described. Indeed, I think that both structures have advantages as well as disadvantages. Once again, there is probably much to be said for healthy competition.

42. This is a real issue, which not even the prestige of MIT, Harvard, or Stanford can overcome readily. Senior academic leaders repeatedly expressed doubts about the wisdom of offering fully prepackaged courses to their students, citing a desire to “brand” courses as their own in order to preserve institutional identity. Of course, this problem would be alleviated greatly if established institutions were to grant credit to students who had earned certificates of accomplishment from MOOCs. But this is a challenging prospect in the case of most four-year institutions, at least without further testing by the institutions themselves or some other third-party method of certifying both the content of the course and the achievements of students taking the course. The collaboration between the ACE and Coursera (mentioned earlier) is encouraging in suggesting a path toward assessing both the credit-worthiness of online offerings and the success of students in passing these courses. Two-year institutions may be more likely than four-year institutions to move in this direction.

43. See Steven Kolowich, “How ‘Open’ Are MOOCs?” Inside Higher Ed, November 8, 2012, for a discussion of the extent to which outsiders are allowed to use MOOC content under various MOOCs’ terms of service.

44. For a discussion of Coursera’s thinking with respect to potential business models, including candid comments by Coursera’s co-founders (Daphne Koller and Andrew Ng), see Young, “Inside the Coursera Contract.” While Coursera’s courses are currently free to students, this may not always be the case; should Coursera start to charge for its courses, colleges that enter into contracts with Coursera might receive a portion of the revenues from those fees. (See Daphne Koller and Andrew Ng, “Log On and Learn: The Promise of Access in Online Education,” Forbes, September 19, 2012.)

With respect to other prominent MOOCs, the leaders of edX have said that, in the near future, they will offer certificates to those who complete its courses for a “modest fee”; the extent to which students will be willing to pay for this certificate, however, remains to be seen. (See “Frequently Asked Questions,” edX, www.edx.org/faq.) In addition, see Katherine Mangan, “Massive Excitement about Online Courses,” Chronicle of Higher Education, October 1, 2012.

45. At his installation, MIT’s new president, L. Rafael Reif, spoke explicitly about the need to address cost issues. (L. Rafael Reif, “Inaugural Address as Prepared for Delivery,” Massachusetts Institute of Technology, Cambridge, MA, September 21, 2012, http://president.mit.edu/speeches-writing/inaugural-address.) As Stanford economics professor Caroline Hoxby has observed, it is surprising that many university leaders fail to analyze the cost and revenue implications of approaches to online learning before they invest in them (personal communication, September 21, 2012).

46. Thus, in the case of Khan Academy, it is hard not to wonder about the viability of Salman Khan’s pronouncement: “Our mission statement is a free world-class education for anyone anywhere.” (See “Changing the Economics of Education,” interview by Walt Mossberg with John Hennessy and Salman Khan, Wall Street Journal, June 4, 2012.)

47. For example, among the learners who completed the circuits and electronics MOOC and who agreed to take a demographic survey, 96 percent said they had previously taken calculus, and about 70 percent had at least a bachelor’s degree. This may not be surprising, given that the prerequisites for this course included (according to the course’s website) an Advanced Placement–level physics course in electricity and magnetism, “basic” calculus and linear algebra, and “some background in differential equations.”

A survey of students in Coursera’s machine learning course revealed a somewhat different population, although this survey was not limited to learners who completed the course, and the main prerequisite for the machine learning course—a basic knowledge of programming—was, arguably, less restrictive than the prerequisites for the circuits and electronics course. The machine learning course survey revealed that among the more than 14,000 respondents, working professionals exceeded students, and among the students taking the course, the majority already had bachelor’s degrees. (More specifically, 41 percent of all respondents said they were currently employed in the software industry, 9 percent said they were professionals employed in non-software areas of the computing and information technology industries, and 2.5 percent said they were employed in an industry other than technology. By contrast, about 20 percent of the respondents were graduate students, 11.6 percent were undergraduates, and 1 percent were enrolled in school at the K–12 level. See Steve Kolowich, “The MOOC Survivors,” Inside Higher Ed, September 12, 2012; Steve Kolowich, “Who Takes MOOCs?” Inside Higher Ed, June 5, 2012.)

48. A recent New York Times article offers a useful overview of MOOC developers’ search for ways to turn their courses’ popularity into reliable sources of revenue. (Tamar Lewin, “Students Rush to Web Classes, but Profits May Be Much Later,” New York Times, January 6, 2013.)

49. Someone asked, on Twitter, “Why would Antioch pay for something that is free?” The answer is that Coursera’s terms of service prohibit (quite properly in my view) the use of Coursera’s MOOCs for anything but informal education. These restrictions, which are characteristic of other MOOCs as well as Coursera, complicate the ability of colleges and universities to incorporate MOOC offerings in their regular for-credit curricula. (See Kolowich,“How ‘Open’ Are MOOCs?”)

Readers interested in this subject should also see the op-ed piece in the Wall Street Journal on October 2, 2012 (in the online edition), by the new president of MIT, L. Rafael Reif, titled “What Campuses Can Learn from Online Teaching.” Reif raises the possibility that “online education may improve the financial model of residential education.” He envisions a world in which “a university’s courses can be offered online for small fees to people around the world,” and he suggests that “we might arrive at a sweet spot where high numbers of online learners are getting extremely good value for their fees, and the university that creates the content is using those fees to serve the mission of the university as a whole—part of which is to make education, on and off campus, affordable.” In fact, Reif is not suggesting that everyone who registers for such a course be charged—which would contradict the notion that such courses are free—but that those who seek credentials certifying that they have mastered the content be charged a small fee for such certification. However, no one knows how such a plan would work in practice, and I continue to suspect that fees for institutional use of online courses may be a more viable way of addressing the various economic issues that confront us. Of course, it is conceivable that both individuals and institutions could be charged. Right now, all of this is highly speculative.

50. Bacow et al., Barriers to Adoption, 19–20.

51. Ibid., 29. One real problem is that ranking methodologies often focus on simple student-faculty ratios as measures of quality, and leaders at both private and public institutions understandably worry that online instruction may affect such ratios even if it does not affect learning outcomes. Equally deplorable is the exaggerated emphasis on SAT/ACT scores, which can lead to unwise admission decisions.

52. It is sobering to note the findings in a survey released in December 2012 by the Association of Governing Boards (AGB). While a majority of the members of boards who responded said that higher education in general costs too much, an even higher percentage (62 percent) said that “their institution costs what it should.” (Sara Hebel, “Board Members Say College Costs Too Much, but Not at Their Institution,” Chronicle of Higher Education, December 13, 2012.) More awareness of the need to control costs “at home” is needed.

Another point worth reinforcing is the absence, from much of the discussion of the MOOCs’ potential, of any emphasis on using such online offerings to save serious money on traditional campuses. The University of Texas System, which is now part of edX, is an exception to this generalization. The UT System is clearly interested in using MOOCs to “get more students through college more quickly and for less money.” (See Steve Kolowich, “UT Hopes to Award Credit for Gateway edX Courses,” Inside Higher Ed, October 16, 2012.) It remains to be seen, however, to what extent institutions in the UT System will, in fact, give credit for edX courses on an “as-is” basis, and to what extent (and at what cost) they will want to modify these courses to make course material developed largely for populations of students at Harvard and MIT useful to the much different student population in the UT System.

53. A majority (57 percent) of adults said they believed the U.S. higher education system was doing a fair or poor job of providing value for the money spent by students and their families, according to a spring 2011 survey by the Pew Research Center of more than two thousand adults. The proportion of adults agreeing that most people can afford to pay for college also fell from 39 percent in 1985 to 22 percent in 2011. (See Paul Taylor et al., “Is College Worth It?” Pew Research Center, May 15, 2011, www.pewsocialtrends.org/2011/05/15/is-college-worth-it/.)

54. An article in Inside Higher Ed, summarizing the results of a survey of faculty members on their attitudes toward the use of technology in education, defined “flipping the classroom” as “banishing the lecture and focusing precious class time on group projects and other forms of active learning.” Inside Higher Ed was one of the two entities that conducted this survey; the other was the Babson Survey Research Group. (Steve Kolowich, “Digital Faculty: Professors and Technology, 2012,” Inside Higher Ed, August 24, 2012.)

55. See Kolowich, “Digital Faculty”; I. Elaine Allen and Jeff Seaman, “Digital Faculty: Professors, Teaching, and Technology” (Babson Park, MA: Babson Survey Research Group and Inside Higher Ed, August 2012), http://babson.qualtrics.com/SE/?SID=SV_bJHd6VpmahG2NGB.

56. I discuss principles of delegation in chapter 2 of William G. Bowen, Lessons Learned: Reflections of a University President (Princeton, NJ: Princeton University Press, 2011). Of course, matters need not be, and should not be, so black and white; there is much to be said for nuance and the application of common sense. But I do understand why Gary C. Fethke and Andrew J. Policano (former interim president of the University of Iowa and former dean of its business school, and current dean of the Paul Merage School of Business at the University of California, Irvine) bemoan an “hourglass” power structure, in which some kinds of authority are exercised solely by the regents at the top, and many other kinds of authority are exercised solely by the faculty, with the administration sometimes caught in the middle. (See Fethke and Policano, Public No More: A New Path to Excellence for America’s Public Universities [Stanford, CA: Stanford University Press, 2012], 217.)

57. Let me give an example of sometimes troubling experiences JSTOR has had in attempting to explain the economic value of access to its digitized collections of scholarly literature. Sometimes (not infrequently) decisions as to whether to subscribe to a JSTOR collection are vested in the library and are thought about solely in terms of comparisons with the value of traditional acquisitions. This is much too limited an approach in that it does not take account of longer-term space savings (which are, one librarian told us, “no concern of his,” since they come out of someone else’s budget) or more complicated notions of the changing role of the library.

58. The course reinvention initiatives sponsored by the National Center for Academic Transformation (NCAT) generally focused on departments and thus led to situations in which the departments concerned captured all the savings. (See Ben Miller, “The Course of Innovation: Using Technology to Transform Higher Education,” Education Sector, May 2010, www.icuf.org/new​development​/wp-content/up​loads/2010/06​/Technology-in-Higher-Ed.pdf, for a summary of several NCAT transformations.) It is easy to understand why this approach makes sense in terms of the desire to enlist support from faculty. But it does not address the more fundamental issue of saving costs for the institution at large or for students and families. Derek Bok has observed that “it will take rather adroit and firm leadership to capture the savings for institutions rather than have them disappear down the black hole of somewhat lower demands on the time of professors who teach these courses” (personal communication, October 30, 2012).

59. See American Association of University Professors, 1940 Statement of Principles on Academic Freedom and Tenure with 1970 Interpretive Comments (January 1990), available at www.aaup.org/aaup/pubsres/policydocs/contents/1940statement.htm; American Association of University Professors, The Rights and Responsibilities of Universities and Their Faculties (March 24, 1953).

60. Indeed, mistakenly equating academic freedom with unbridled faculty discretion, including control over methods of delivering classroom instruction, is dangerous and even self-defeating. The use of overly sweeping academic-freedom arguments to block reasonable efforts to innovate puts the core principles of academic freedom at risk.

61. A recent article in the Economist predicts that the unbundling prompted by the emergence and proliferation of MOOCs and other forms of educational technology will be coupled with increased specialization, with some institutions “perhaps dropping indifferent lecturing or teaching to concentrate on something else, such as brilliantly set and marked examinations.” Another likely result of unbundling is collaboration across institutions that recognize the value of pooling resources for purposes like offering common introductory-level undergraduate courses. (“Learning New Lessons,” Economist, December 22, 2012.)

62. Joseph E. Aoun, “A Shakeup of Higher Education,” Boston Globe, November 17, 2012.

63. Clay Shirky, “Napster, Udacity, and the Academy,” blog post, November 12, 2012, www.shirky.com/weblog/2012/11/napster-udacity-and-the-academy/.

64. Not all institutions share this perspective; some top-tier institutions do not see a compelling reason to change their current operations. For example, according to the Economist, Oxford and Cambridge remain “determinedly unruffled” by the emergence of MOOCs: “Oxford says that MOOCs ‘will not prompt it to change anything,’ adding that it ‘does not see them as revolutionary in anything other than scale.’ Cambridge even says it is ‘nonsense’ to see MOOCs as a rival; it is ‘not in the business of online education.’” While MOOCs are still a relatively new development, and may not currently pose a formidable threat to Oxbridge, steadfast resistance to adaptation in light of new technology may prove to be shortsighted. (“Learning New Lessons.”)

65. Kevin M. Guthrie, president of ITHAKA, has written a thought-provoking paper on this general subject: “You Can Run But You Can’t Hide: Colleges and the Forces of the Internet” (delivered at the 2012 Lafayette Conference on the Future of Liberal Arts Colleges, Lafayette College, Easton, PA, and to be published in Education for an Uncertain World: The Future of the Liberal Arts College in the Age of Technology, Globalism, and Economic Challenge, eds. Rebecca Chopp, Susan Frost, and Daniel H. Weiss [Johns Hopkins University Press, forthcoming]).

66. For examples of the power of inspired teaching in intimate settings, see William G. Bowen, “More to Hope than to Fear: The Future of the Liberal Arts College,” presentation at the Lafayette Conference on the Future of Liberal Arts Colleges, Easton, PA, April 10, 2012, to appear in Chopp, Frost, and Weiss, Education for an Uncertain World.

We should not, however, exaggerate the distinction between what it is possible to achieve through inspired person-to-person exchanges and the rich interactions that are now possible (though by no means guaranteed) through online teaching. Princeton sociology professor Mitchell Duneier quotes a comment from one student who took the Coursera course that Duneier taught earlier this year to 40,000 learners in 113 countries: “It has been an incredible experience for me, one that has not only taught me sociology, but the ways other cultures think, feel, and respond. I have many new ‘friends’ via this class.” (Duneier, “Teaching to the World from Central New Jersey.”) This language is not all that different from what one might expect to hear from a student in a traditional classroom setting—and, of course, the students in this online course represented a wider range of perspectives than one could find in even the most diverse in-person setting. But then we also have to recognize that not all of Duneier’s online students may have had this kind of experience.

67. Indeed, Stanford’s medical school, with assistance from the Khan Academy, is experimenting with posting video lectures online for some of its classes. Instead of the professor’s conveying the course content via live lectures (which, according to professors’ estimates, about 70 percent of students do not attend), students watch the video lectures online before coming to class, and the professor devotes the in-person meetings to more interactive activities—such as meetings with patients, debates among students, and group exercises—that require students to apply what they learned from the video lectures. The professor leading this initiative has expressed hope that these online materials can be shared someday with similar medical schools, which teach essentially the same first- and second-year courses, as well as with medical professionals, both in the United States and around the world. (See “Medical Education at Stanford Gets More Interactive by Going Online,” Stanford University, September 28, 2012, http://news.stanford.edu/news/2012/september/medical-education-online-092812.html.)

68. Quoted in John Hennessy, “Prepared Remarks for Commencement 2012,” Stanford University, Stanford, CA, June 17, 2012, http://news​.stan​ford​.edu​/news​/2012​/june​/hennessy-commencement-remarks-061712.html.

69. Upon completing a Coursera MOOC on health care and the Affordable Care Act, Ursinus College politics professor Jonathan Marks wrote, “I am optimistic that this kind of MOOC will not eat my job because it and I are not really in the same business.” Furthermore, he argued, MOOC providers will never replace traditional undergraduate institutions, because the “strongest” examples of the latter have “faculty and staff who have deliberated about” how to address students’ “need to discover and bring to completion their best mature selves,” and who “marshal the resources of the college, inside and outside of the classrooms, to fulfill their missions.” By contrast, a typical MOOC provider offers “classes and teachers united by nothing apart from its platform to students who are expected to know what they want and to pursue it with minimal guidance.” (See Jonathan Marks, “Who’s Afraid of the Big Bad Disruption?” Inside Higher Ed, November 18, 2012.)

Marks is right to stress the importance of institutions’ finding ways to teach students less tangible skills such as good judgment. Also, one can agree that “the capacity to decide what to think or how to act in areas, like health policy, where no formula can generate the right answer” may not lend itself especially well to being taught in a MOOC format. However, agreeing with this proposition does not negate the ability of MOOCs to impart useful knowledge of many other kinds to students in credit-bearing capacities. It is a mistake to be imprisoned in either-or thinking. The relevant question isn’t whether a Coursera course will end the career of political science professors like Marks, or whether it will be able to offer the same kind of guidance as a more intimate face-to-face setting, but rather how to build portfolios of offerings best suited to different student populations and different settings.

70. To this Hutchins added, “Hard intellectual work is doubtless the best foundation of character, for without the intellectual virtues, the moral sense rests on habit and precept alone.” (Robert Maynard Hutchins, No Friendly Voice [Chicago: University of Chicago Press, 1968], 93.)

71. Jeff Bezos, “We Are What We Choose,” baccalaureate address, Princeton University, May 30, 2010, www.princeton.edu​/main​/news​/archive​/S27​/52​/51O99/index.xml.

72. The rush to participate in the “MOOC movement” no matter what (lest one be left behind) is a bit frightening. As the former acting president of Purdue University, Tim Sands, said at a discussion about online learning: “There’s talk that if you’re not in the game, you’re going to miss the wave.” (Elena Sparger, “President Sands and Panel Discuss Online Courses at President’s Forum,” Purdue Exponent, October 2, 2012.) Also see the highly publicized debate surrounding the (temporary) ouster of the president of the University of Virginia, and the conversation about the future of MOOCs at a recent conference at Cornell University. (Andrew Rice, “Anatomy of a Campus Coup,” New York Times, September 16, 2012, www.nytimes.com​/2012​/09​/16​/magazine​/teresa-sullivan-uva-ouster.html?pagewanted=all&_r=0; Akane Otani, “Cornell Professors Debate Future of ‘Massive’ Online Classes at University,” Cornell Daily Sun, September 28, 2012.)

To exaggerate the dangers, but perhaps only slightly, consider this comment by Lawrence S. Bacow, former president of Tufts, after he had listened to one of the many discussions of what some see as the impending transformation of American higher education: “If I went to my board at Tufts and told them that I was purchasing an enterprise software system on which the entire revenue stream depended, and that the vendor had been in business for less than a year and had no track record, revenue, or business model, I suspect I would have been fired. Yet some boards are forcing their presidents to contemplate hugely consequential changes in strategy based on six months’ worth of experience from a handful of MOOCs.” (Lawrence S. Bacow, personal communication, October 2, 2012.)

73. For an excellent example of the time it takes to do good work in this field, see the account of Neil Heffernan’s work on tutoring models cited in Annie Murphy Paul, “The Machines Are Taking Over,” New York Times, September 14, 2012. Heffernan has been working for seventeen years on what appears to be a highly promising way of using machine-guided tutoring to teach math. James Kemple, executive director of the Research Alliance for New York City Schools, also argues persuasively that “effective, sustainable reform requires persistence and adaptation, which must be informed by the accumulation of evidence, over time, about what seems to be working and what doesn’t.” (See James Kemple, “Math Innovation Requires Patience,” New York Times, SchoolBook blog, September 21, 2012, www.schoolbook.org/2012/09/19/innovation-in-math-instruction-deserves-patience-and-persistence.) Kemple is critical of two of the original School of One schools for deciding to discontinue the School of One program before they had an opportunity to incorporate ongoing improvement efforts and assess them. (School of One is a middle-school mathematics program used in some New York City public schools that emphasizes the use of technology to allow for more individualized learning.)

74. See Miller, “The Course of Innovation.”

75. See I. Elaine Allen and Jeff Seaman, Changing Course: Ten Years of Tracking Online Education in the United States (Babson Park, MA: Babson Survey Research Group, 2013), 17. The authors of this report, which was put together by the Babson Survey Research Group in partnership with the Sloan Consortium and Pearson, differentiate between traditional, “web-facilitated,” hybrid, and online courses on the basis of the proportion of content delivered face-to-face versus online (according to survey respondents); “online courses,” by the authors’ definition, are those in which at least 80 percent of the course content is delivered online.

76. The Open University’s website is www.open.ac.uk. David L. Kirp chronicles the development of OU in chapter 10 of Shakespeare, Einstein, and the Bottom Line: The History of Higher Education (Cambridge, MA: Harvard University Press, 2003). See also “Europe Starts to Embrace Online Education,” Science 2.0, September 28, 2012, www.science20.com/news_articles/europe_starts_embrace_online_education-94485.

77. See Indiana University, “Indiana University Announces IU Online, A Major New Online Initiative,” September 5, 2012. The pressures on universities to join the MOOC movement are illustrated well by a recent forum at Cornell University; see Otani, “Cornell Professors Debate Future of ‘Massive’ Online Classes at University.”

78. See Thille and Smith, “Learning Unbound,” 31–38, for a description of the features of the Open Learning Initiative courses. Also see chapter 4 of Walsh, Unlocking the Gates, for a description of the development of the Open Learning Initiative.

79. While the examples provided here generally involve prominent institutions, it is not only the big-name universities that have shown an interest in MOOCs. For example, the University of Maine at Presque Isle has announced an initiative that will “allow learners of all ages to participate in online college courses for free, as long as they aren’t seeking college credit.” Initially, the project will involve a slate of English courses, and care is being taken to avoid copyright issues. (See Jen Lynds, “UMPI OpenU expands access to college-level courses for free,” Bangor Daily News, August 16, 2012, http://bangor​daily​news.com​/2012​/08​/16​/news​/aroostook​/umpi-openu-expands-access-to-college-level-courses-for-free/.)

80. See Hannah Seligson, “10 Universities to Form Consortium for Online Courses,” New York Times, November 15, 2012.