Epilogue

I began this book with an observation by Richard Feynman, one of the twentieth century's most revered scientists.

It doesn't matter how beautiful your theory is, it doesn't matter

how smart you are. If it doesn't agree with experiment, it's wrong.

It seems to me that I am unlikely to go wrong ending with some allied observations.

The title of this book was chosen purposely to be evocative; when we make guesses about how to improve any complex process—and education certainly falls into that category—we ought to use evidence to guide our guesswork. Such guesses I would class as “educated.” Then after we make those guesses we must subject them to careful scrutiny, both empirical and logical. And if the outcomes observed, after suitable checking to assure that all intervening steps are correct, are not what we expected, the guess was wrong. This emphasizes one consequence of the decision to use evidence as the deciding factor—evidence, not expert opinion. Once again, Richard Feynman:

Science is the belief in the ignorance of experts.

I recognize that the decision to base actions on evidence is a tough one. In the last year we have seen how people armed with anecdotes, but not data, attacked evidence-based advice about who should have mammograms. Indeed many of the anecdotes relied on emotional counterfactual statements for their validity: “If my sister had not had a mammogram, she would not be alive today” or “If she had a mammogram, she would be alive today.” The fact that such logically weak argumentation yielded almost immediate equivocation from government officials attests to how difficult it is to use the logical weight of evidence over the emotional power of anecdote. We must always remember that the plural of anecdote is not data.

The battle opposing emotion and anecdote against logic and evidence makes for fine theater—witness how popular were the inevitable showdowns between Mr. Spock's Vulcan logic and Captain Kirk's human emotion. I have argued that for setting policy or for making triage decisions in the face of scarce resources, we do far better using evidence. My role in this parallels Spock's when he explained, “Nowhere am I so desperately needed as among a shipload of illogical humans.”¹

This book is dedicated to demonstrating how evidence can be used to cast light on important issues, many of which are still under consideration. My concern, which led to the preparation of this book, was that despite counterevidence, various proposals for change in educational practice now gaining momentum are likely to be put into practice. I hope that the results of the mini-experiment provided by Bowdoin's experience in making the SAT optional will lead to a larger-scale examination of that logically and empirically flawed proposal. The formal adoption of a policy to evaluate teachers through the average change in test scores of their students was finalized in the Tennessee legislature in April 2010, and seems close to implementation in many other states. This idea sounds sensible if said quickly, but when examined more carefully we find that it is not ready for prime time. Some version might be useful in the future, but not yet.

All of this brings me to the topic of hubris. On these pages I have railed against those who advocate changes to the status quo without the support of empirical evidence to show that the changes would improve matters. This represents two different kinds of hubris: theirs in thinking that they are wiser than their predecessors whose work has yielded the current system, and mine in suggesting that their ideas are wrongheaded.

Their hubris first. We are not any smarter than those who have come before us. If we can make improvements, it is only because we have the benefit of additional experience—more evidence. In any complex operation, whether it is a manufacturing process or an educational system, there is always room for improvement. Yet how are we to reach the future? Experience has taught us a great deal about what kinds of optimization methods work in complex systems and what kinds do not. An almost surefire path to failure is to convene a blue-ribbon committee with a title like “Education 2020” whose mandate is to ponder existing problems and come out with recommendations for the future system. It doesn't work because even all-stars aren't that smart. What does work is the implementation of constant experimentation, in which small changes² are made to the process. Then the effects of the changes are assessed. If the process improves, the size of the changes is increased and the outcome monitored. If the process gets worse, the changes are reversed and another variable is manipulated. Gradually, the entire complex process moves toward optimization. If we follow this approach, when the future arrives, the system of the future is there to greet it. Critical to such an approach working are well-defined and well-measured outcome variables so that we know what “improved” means. We also need to understand enough about the process to be able to select what are the most likely parameters, those whose change would affect the outcome variable, and so guide the experimental manipulations. It is on these two aspects that blue-ribbon panels may be useful—and to lend political legitimacy to the entire enterprise.

Underlying the success of such an approach is the epistemological belief that evidence, and not the power of authority or faith, must guide change.

Now my hubris in making these suggestions: To write a book requires the sustained belief that your thoughts and findings are of interest and value to others. The sin of pride must be the one common trait that runs through all authors. In my own defense note that my recommendations are not based on any misplaced notion of my own special genius, but rather on the much more pedestrian collecting of facts over a lifetime of work. Thus, the frequency with which I write books has increased with my years.

Marcel Proust likened aging to being “perched upon living stilts that keep on growing”—thus we can see further as we age, but passage is increasingly wobbly. This book is a testament to that process.

¹ Spock in “I, Mudd.”

² “Small changes” can be small in the size of the change (like increasing the school year by a few days) or small in the number units involved (trying it out at just a few carefully chosen schools).