William G. Bowen’s Responses to Discussion Session Comments by Andrew Delbanco and Daphne Koller

IT IS EVIDENT that Daphne is a true believer! And when I referred earlier to the missionary spirit, here you see it incarnate, and it is wonderful to see. What I think is especially admirable is that in Daphne, the missionary spirit is blended with an understanding that you do have to test things and to look for evidence. As I have said, a big barrier to greater acceptance of online learning is the lack of evidence concerning both learning outcomes and cost savings. Assembling evidence takes thought and it takes patience; it is also very, very important.

Let me now turn to Andy’s remarks, which I thought were very good. Your comments, Andy, about the value of provocations remind me to underscore something that I may not have emphasized sufficiently. In our desire to experiment and to try new things, it is important to recognize that the pedagogy best suited to teaching in one field may not be the same pedagogy that is appropriate in a different field. A one-size-fits-all attitude is dangerous because it will lead both to bad teaching methods and to confusion about what is possible. I think there is merit in distinguishing, at least roughly, between fields in which there are understood facts or concepts that need to be acquired—and more discursive topics. What is a confidence interval? There is an answer to that question. That question is different in kind from discussion at a deep level of why the Palestinians and the Israelis have so much trouble getting together. Now that is a tremendously important question, but it is a very different question and requires a different way of thinking. We certainly need provocative teaching in subjects like that.

Now, I am not suggesting that we overdo this distinction; we should not. If you teach concepts well, you can generate a great deal of stimulating discussion, and I am impelled to mention a personal example. Without a doubt, the most powerful learning experience I ever had came when I was a frightened beginning graduate student in economics, taking William J. Baumol’s course in economic theory. Professor Baumol used a book by J. R. Hicks called Value and Capital, probably one of the worst-written books ever produced, and I always suspected that Professor Baumol chose it in part for that reason. If I remember correctly, we covered in a semester-long course thirty-five pages. Thirty-five pages! Here is the way the course was taught. We would open the book and look at this incomprehensible, inscrutable paragraph, and Baumol would say, “All right, the assignment for next week is that each of you will go home and think, and write—write clearly—two pages explaining cogently what this paragraph actually means.” I remember going back to my room and puzzling, puzzling, before going to bed—and all of a sudden waking up in the middle of the night and saying to myself: “I understand!” I then raced to my desk and wrote down what I thought was an insight before I lost it. What I learned from that experience was that I might not be as fast as others in the class in grasping every point, but, given time and persistence, I could actually figure many things out. I learned through that experience to have a kind of quiet confidence in what it was possible for me to achieve. That was a lesson of life, if you will, that was independent of the content. Particularly at advanced levels, we have to have the time, opportunity, and resources needed to permit that kind of life-changing experience—which is, to be sure, more likely to be provided by truly brilliant teachers such as Professor Baumol than by mere mortals.

Now, Baumol’s approach did not work for everybody. I remember a classmate in that same course who, when asked to go to the blackboard (as Baumol always asked students to do), to take a piece of chalk, to draw a demand curve, and then to explain in a simple sentence what a demand curve is, was paralyzed. He finally just left the course and left economics forever—which was probably a good outcome all around. This is only to say, again, that we have to avoid all-or-nothing mindsets. How you teach statistics is different from the way you teach Melville. It should be different, and how you teach one set of students at one stage in their lives may be very different from how you teach others at different stages in their lives. This is why I argue for a portfolio approach to curricular development, for finding mixes of instructional styles, and mixes of pedagogies, that allow different kinds of learning to occur.

I should also take up a point that Daphne made with which I agree. In revising these lectures for publication, I will make clearer my sense that the equivalent learning outcomes we found in our empirical study of the CMU statistics course represent a kind of baseline. That is how these findings should be viewed. The teachers in the hybrid-online sections of the course we tested were inexperienced: they had never done anything like this before. Some of them seemed nearly baffled by the pedagogy. Some were essentially dragged, kicking and screaming, to this assignment. Could we do better? Certainly. Brit Kirwan, the enormously capable chancellor of the University System of Maryland, which was a participant in our project, is firmly convinced of that. Yes, learning outcomes will improve.

There were two big deficiencies in the very good CMU statistics course that we tested, both of which can be addressed. One was a lack of “fun.” A friend of mine at CUNY explained the problem this way: “The course was just not fun. It was not addictive; it did not have any Disney-like features.” Then there is the issue of customization, which is a tricky and complicated one. There is no doubt in my mind that a limitation of the course we tested was that it really could not be customized to meet “local” needs.

Let me mention another aspect of online teaching that, again, Daphne mentioned: adaptive learning. I have learned from Daphne how hard it is to incorporate truly adaptive learning in online courses. One of the great strengths of the Carnegie Mellon statistics course that we tested was that it was an adaptive learning course. If a student got something wrong, he or she could push a “hint” button and receive a well-thought-out suggestion as to how to do better. Such suggestions were based on evidence concerning the learning experiences of many students. If the student tried again and still got it wrong, pushing “hint” again would display another hint, and then another. When the student finally solved the problem, he or she was given another problem that was similar. That kind of machine-guided learning model works for certain content, probably not for Melville. It has great potential, especially in certain fields. But Daphne has helped me understand that this approach is really hard: it requires masses of data and discipline-specific knowledge. I think the pursuit of adaptive learning is on Coursera’s radar screen, but it is down the road.

In terms of productivity, and numerators and denominators of productivity ratios, I intend in my revision of these lectures to do a better job of parsing out the elements of potential productivity gains. I agree that there is a huge potential for improving cost-effective learning outcomes by reducing time-to-degree and increasing completion rates. If, in fact, students can get through gateway courses faster, if they can present credentials that allow them to move through the system more rapidly, there could be substantial improvements in system-wide productivity.

I think in many ways the most challenging issue is one Andy identified: how to manage to keep the right value orientation and the real learning aspect of education in large, resource-strapped public universities. For many of these universities, the real challenge is how best to take some of the resources that can be saved by an appropriate use of machine-guided learning (in fields in which it makes sense), and allow those savings to be used to do the other kinds of teaching that are so important. This observation takes us back to the whole set of issues involving governance, decision-making, and resource allocation, writ large. These are issues that the academy really needs to think through because compartmentalized decision-making, the “every small tub on its own bottom” approach, is not going to get us where we need to go.