Discussion by Daphne Koller

GOOD MORNING. That will be a very tough act to follow. I am going to try to respond to a few, not all, of Professor Delbanco’s points in the comments that I will make, and hopefully, we will tackle some of the rest in the discussion. Let me begin at a very similar point to Professor Delbanco’s talk. Professor Bowen started his second talk saying that he was initially skeptical about the value of online learning but has come to believe that perhaps now is the time when this technology will actually come to fruition and help us come up with a transformed educational experience. A question that I get asked often is what is it that makes this different from the multiple previous attempts, the multiple notorious failures—such as the Fathom project—that did not succeed in transforming the shape of education. Here we can also include the TV-based efforts, which I was not even aware of because they long predate me. There are a lot of little pieces which I’ll come to in a moment, but if you had to put your finger on one thing, it is the recognition that this new form of teaching is really a new educational paradigm. That is, it is not and should not be taking in-class instruction and trying to replicate it in an online medium.

By thinking about this in a completely new way, we can release ourselves from the shackles that we have gotten used to in the context of in-class teaching. And, while perhaps losing some of the benefits of face-to-face instruction, we can gain in other ways. By contrast, in the other approach you lose some of the benefits of in-class instruction without gaining anything, or very little. I think that this realization has led to a lot of the specific innovations that have really contributed to the success that we have seen in this new generation of online learning platforms, ours as well as others in this space. Perhaps most important, there’s the fact that interaction of at least two different types needs to be built into the learning process.

First, there is interaction of the student with the material. We should avoid a passive one-way communication where the students are sitting there with the information flowing unidirectionally toward them, as was the case in the TV-based or in the Fathom effort. Rather, we should aim for a constant interaction with the computer, which is now capable of “talking back” to the students.

Equally or perhaps even more important is the fact that students can actually rub minds with each other via the computer. Now, this is something that to the people of my generation is still a little bit foreign, but if you talk to the kids of today, the eighteen-year-olds, they actually prefer to text each other rather than to talk to each other on the phone or even get together for coffee. That kind of interaction via an electronic medium is something that is really built into the culture now. It has been built into us by Facebook, by Twitter, by text messaging. An important lesson from this generation of students is that we should not presume, because of the way that we were brought up, that rubbing minds can only happen in face-to-face interaction. The rubbing of minds via digital media is a key to the success of this new generation of teaching platforms. Many of our students who have taken some of the courses on the platform tell us that in fact, this is a far more interactive experience than they have ever had in face-to-face instruction.

Now, this is not the case, I am sure, for Professor Delbanco’s ten-person seminar, which is a truly unique experience that any of us would be privileged to participate in. But consider most of the kids in large universities, who are shoveled into large auditoriums with three hundred people and the professor standing down there and lecturing at them. And then they go off to their dorm rooms and do their homework. These kids get a lot more interaction with both the material and their fellow students by being on this online system where they interact with each other on a constant basis. If I had known that we were going to hear this very inspiring quote from Professor Delbanco at the end of his talk, I could have given multiple similar quotes from many of our tens of thousands of students. In particular, I could probably have come up with hundreds from our class of 33,000 students who are taking Al Filreis’s Modern and Contemporary American Poetry class. Al Filreis is the head of the U. Penn Writing Center. His typical class there is a small seminar, much like Professor Delbanco’s. But he has 33,000 students in his online class. And he gets e-mails from students that are very similar to what we just heard, about how these interactions have opened these students’ minds to a new way of looking at poetry, have allowed them to interact with their peers in a way that is centered on the course material but also transcends geographical boundaries, and have provided them with really profound communication with large groups of people that they would never, in the ordinary course of events, have the opportunity to meet.

I wanted to start off with that comment because I think it is important to respond to some of the justified caution in the notes that we have heard. In particular, it is important not to come in with too strong of a set of preconceptions about what online technology can and cannot do in terms of rubbing minds and in terms of inspiring groups of people who would never have the opportunity of taking Professor Delbanco’s class.

With that said, I am going to switch to a different topic, which follows on one of the most important observations made at the very start of Professor Bowen’s first lecture, the key metric of productivity. That is, the ratio of outputs relative to inputs or outcomes relative to costs. Now, as Professor Bowen eloquently argues, in the context of education, we cannot actually measure either the numerator or the denominator; in fact, we do not even know what we are measuring, which clearly limits our ability to measure productivity and try to improve it. This is one place, among many, where Professor Bowen was a true leader in the field; he was the first to highlight the notion of completions rather than enrollments as a key measure of productivity. He also focused on learning outcomes as being a key measure of productivity: the understanding of the material as well as the ability to use the material in new and innovative ways. And then, if we want to move beyond that, there are also longer-term outcomes, such as the ability of a student graduating with a degree to earn a good living and, even more important, to have high job satisfaction. That is, we want our graduating students to be proud of what they do when they go to work every day. All of these, of course, are even harder to measure than the costs.

On my optimistic days, I would argue that one of the great promises of technology in education is that if done right—and I think that qualifier is important—it can enact significant change for the better in both the numerator and the denominator of this equation. I would like to talk about each of those pieces. Let me start with one of the key aspects of the second talk, which is the seminal ITHAKA study on the gains of online methodologies and improving efficiencies within academic institutions. This was a landmark study because it was the first rigorous case-control study that actually compared learning outcomes between blended and face-to-face learning. And certainly there were a lot of promising aspects of that study. The first is to demonstrate conclusively (although I will come back to that) that there was no difference in learning outcomes in these two populations. That means that the numerator was left unchanged, and the denominator was potentially improved in at least two ways. First was a decrease of 25 percent in the amount of time spent by students, so that was an improvement in the cost, at least the cost to the students. Second, to a certain extent, there was an analysis suggesting that we could achieve a reduction in the investment required by the institution in terms of instructor and TA effort.

The reductions, however, were relatively modest, and it seems like no matter how optimistic we are going to be, we are likely to see a reduction in costs of on-campus instruction by a factor of about two or three at best. Of course, as Professor Bowen correctly points out, it is hard to judge steady-state improvements in costs over a long time frame from a single instance because obviously, efficiency will improve as we learn to do blended learning better. But still, there are many reasons to be concerned about potential lack of reductions in costs, whether because of the caps on section sizes or because of the need, whether real or perceived, by instructors at institutions to customize the curriculum for their own students. And, therefore, if there is going to be productivity improvement in academic institutions, maybe it is not going to be in the denominator.

And so where is the potential in improving productivity in this process? I would argue that perhaps it comes in the numerator. This comes back to what is, to me, one of the disappointing aspects of the study, which is that there is no difference in learning outcomes. Blended learning was not better than face-to-face instruction. Now, some of you might be sitting there saying, why would you expect it to be any better? After all, common wisdom has it that it is only going to hurt our students to move to an online format, so why would you be disappointed by something that showed no change? For me, there are two arguments as to why blended learning might be better, although, obviously, these remain to be proven.

The first comes back to the beginning of my comments, in which I argue that the current online format is actually considerably different from many other online attempts in the past and is considerably more interactive and engaging than a standard lecture. Not more engaging than sitting in a seminar of ten or fifteen people, but when compared to the three-hundred-person auditorium, it actually is a much more interactive experience for the students. There is an incredible amount of educational research by renowned experts in the field that shows that this type of interaction and adaptation lead to better learning outcomes. Some improvements arise from the fact that students are constantly tested on the material. Others arise from the ability to enforce mastery in a given topic before the students move on to the next. These are strategies that online methodologies can help us implement but perhaps were not employed to their fullest extent in our attempts up to now.

The second reason for hope is that moving lecturing out of the classroom supports active learning in the classroom—the ability for an instructor to engage in meaningful ways with the students and for students to engage in meaningful ways with the other students. Now, this particular ITHAKA study did not demonstrate that the blended-learning format gave rise to that type of improvement. However, there are many studies by Eric Mazur, Carl Wieman, and others that have demonstrated significant gains to active learning in the classroom. So why did these benefits not manifest here? One answer is that, for some reason, they don’t arise in a blended-learning format, even if they do when students are asked to read material outside the class. That answer doesn’t seem all that plausible. A second argument draws on Professor Bowen’s own argument, and I am going to quote: “A fundamental problem, cutting across all types of online offerings, is that contemporaneous comparisons of the costs of traditional modes of teaching and of newly instituted online pedagogies are nearly useless in projecting steady-state savings—or, worse yet, highly misleading. The reason is that the costs of doing almost anything for the first time are very different from the costs of doing the same thing numerous times.” This was an argument as to why we did not see significant improvements in the costs; but maybe it is also the reason why we did not see significant improvements in learning outcomes. Because I can tell you, as someone who has flipped my own classroom for three years now, promoting active learning in the classroom is hard. It is not the kind of thing that we were trained to do. We were trained to stand and orate, and if you have been doing it, like I have, for over fifteen years, you tend to become reasonably good at it. Whereas the kind of pedagogy that is required for active learning in the classroom is considerably different. And until you learn that skill, you might not see the better learning outcomes that we hope to see from this new pedagogical intervention of blended learning. So perhaps the fact that we did not see an improvement is not because it could not exist but because, like potential cost reductions, it does not exist yet. This, I think, is one of the most exciting opportunities here.

The second aspect of why I think there is a potential here for significant improvements is the issue of capacity and completion. This is an issue that was briefly mentioned, whereby many community colleges are currently running up against serious capacity constraints. Indeed, in one of the many interesting endnotes to his talk, Professor Bowen points out that a survey by the California community college system shows that as of this summer, more than 472,000 of the system’s 2.4 million students were put on waiting lists for core classes, meaning that they could not make the required progress toward their degree. Enrollment in the California community college system has decreased by close to 500,000 students due to budget cuts, and the number of course sections has been reduced by 24 percent. These reductions give rise to a significant increase in time to completion and presumably also a decrease in completion rates, as people just get tired of waiting around to be admitted into the classes they need in order to complete their degree.

Online learning, by allowing us to increase section sizes and reduce instructor teaching loads, might allow us to increase capacity at community colleges and other institutions, which would allow us to improve, again, the numerator: both increasing completions and reducing completion time to six or maybe, who knows, even four years.

The third potential for improvements is in long-term outcomes, which I briefly alluded to earlier. These online courses have the potential of allowing community colleges as well as other small schools to expand their curricula beyond the expertise and resource constraints of their current instructional staff. They can draw upon courses from Princeton or Stanford or Columbia and offer those to their own students with some additional on-site enhancements, such as blended learning. This expansion would allow students who are attending these institutions to add to their transcripts courses that might make them more eligible for better jobs. For example, in my own discipline, a course in machine learning is an excellent addition to a student’s transcript and can easily make a student eligible for a job in some of the top IT companies. However, such a course can generally be offered only by the top twenty or thirty computer science departments, because of the difficulty of hiring instructors qualified to teach it. If we can offer this class in a much larger set of institutions, we can suddenly make a large number of students considerably more eligible for high-quality positions in companies such as Google or Twitter.

A fourth, even longer-term, opportunity is that of using data and analytics. This is an opportunity that we have yet to execute on, but one with huge potential. We can now collect every click from tens of thousands of students and start analyzing what in our classes is working and what is not: what our students are confused about, and when they figure it out, what they did to reach that understanding. That is something that is going to allow us to improve the way in which we teach our on-campus courses and again, this is not an advantage that you see the very first time that this is done.

A very different opportunity for improvement, one that was not really highlighted in Professor Bowen’s talk, is the ability of these open online courses to be supportive of metrics of success in on-campus teaching even by students who are not enrolled in those institutions. Some of you might know that the Gates foundation put out a request for proposals a few weeks ago for MOOCs in twenty-seven of the highest-enrollment classes in the United States, a list that comprises developmental skills courses (math and English), some gateway classes, and basic general education classes. Those twenty-seven classes account for 20 percent of the units taken by students in the United States. If students could take these classes as MOOCs before they even enrolled in an academic institution, they would be able to come in with a significant number of units already under their belt. That could be a huge win in terms of increasing completion rates, because students who come in with a certain number of credits are much more likely to complete than ones who come in with a blank slate. This could also improve learning outcomes by having students come in considerably better prepared.

Another opportunity of open courses is that they allow the students a free exploration opportunity. People can take courses without financial cost and without risk of a failure on their transcript, which means that students can explore. They can take a little bit of psychology, a little bit of literature, a little bit of math and find out the right fit for them at zero risk before they enter an academic institution. They can also identify the level of challenge to which they can aspire, potentially reducing the mismatch problem that was also mentioned in Professor Bowen’s talk. A student might say, “Hey, I can take and do well at a Stanford class, so maybe I should apply to Stanford after all.” This, I think, is another opportunity for improvements.

Finally, I would like to talk about education outside the boundaries of academic institutions. This is a topic that Professor Bowen explicitly said he was not going to talk about, but I would like to talk about it for just a couple of minutes in conclusion. Education transcends the boundaries of our academic institutions. Most of the students currently taking these open online classes are educated professionals, often with one or more degrees. They want to take these courses sometimes to enhance their credentials for jobs but, in many other cases, just to expand their minds. The people taking modern poetry are probably not doing it to get a better job. We see in these courses a testament to the value of education across the range of disciplines. This observation comes back to one of the points that was made very early in the talk, that of booster shots of education. We no longer live in a world where what we were taught in college twenty years ago is enough to last us for our entire life. The world now changes much too quickly. To continue to have access to the best jobs, or even just to stay informed as world citizens, these booster shots are going to play an important role.

Finally, our discussion so far has been very U.S.-centric. I am going to step outside the United States for a moment and think about the fact that currently two-thirds or more of the students taking these classes are not within the United States. That globalization of education has benefits to us even within the United States. The very fact that our students can interact with people in Kazakhstan and Bangladesh is a way of opening people’s minds to a much more global perspective. Along those lines, Mitch Duneier, a Princeton professor who taught the first humanities MOOC, says that “within three weeks, I had received more feedback on my sociological ideas than I had in a career of teaching, which significantly influenced each of my subsequent lectures and seminars.” He specifically speaks to the global perspective in stating that feedback from thousands of students around the world was fundamental to his perception of his teaching.

Another benefit to opening our doors up to the world is that, again from a selfish perspective, there is unique talent in Mongolia, in Ghana, in Bangladesh—students who achieve perfect scores in some of our most challenging courses. This gives us the opportunity to recruit and identify some of the world’s best talent to come to our institutions and enhance our own talent pool and the mix of students that we teach.

But I would like to conclude with a more unselfish view. We are uniquely privileged in the world in having access to high-quality education. As an example, the average amount invested in the education of a U.S. child by the time they graduate is about $100,000. The average amount invested in the education of an African child is $400—lower by a factor of 250. If we could educate more people, that has the potential of making the world a much better place, because many of the world’s problems can be alleviated by education, including hunger, unemployment, extremism, and even population explosion. One interesting statistic that I have recently seen is that there is a strong inverse correlation between the number of children a woman typically has in a country and the number of years of education for girls in that country. This pattern holds up even after one corrects for culture, for religion, for wealth. Education of women is a way of reducing the population explosion.

The value of education is something that is clearly recognized by some of the world’s poorest people, who are the ones who can benefit the most. In September, I attended the launch of the Education First Initiative in the United Nations, and the former British prime minister Gordon Brown, who is now the special envoy for education at the United Nations, spoke of his travels in Africa. He spoke of talking with people who lived on a cup of food a day, but when asked what they want, they didn’t say more food, they didn’t say better shelter—what they most want is a better education for their children. And so, in our discussion of cost reductions, we shouldn’t forget one place where costs are really important: in allowing us to reduce the marginal costs per student to the point that we can actually provide an education to people who otherwise would never be able to afford it.