by Marc Tucker
There seems to be virtually universal agreement that, once goals are established for improving education systems, the means chosen by policymakers and practitioners for reaching those goals ought to be based on scientific evidence whenever possible. Indeed, in the latest version of the United States’s national basic education legislation (ESSA), states and districts are required to use evidence-based approaches to address the challenges they face. We observe that our national legislators are much better at requiring others to go where the evidence leads than they are at doing this themselves. But they are hardly alone. Most successful national education systems make a determined effort to gather and use the best evidence available more often than those that are less successful, which is one of the reasons their students do so well. But such behavior is unusual, for all kinds of reasons.
First, all of us tend to reject data that does not affirm the way we see the world. Putting it this way makes it sound as though we prefer to be stupid, but that is inaccurate. We are bombarded by data about everything all the time. The only way we can deal with it is to put that data into certain “frames” that experience tells us are powerful explainers of the world around us. Some of these frames are, in effect, metaphors taken from one realm of our lives that we repeatedly apply to others. Not only is this a far more efficient way to process new information than trying to deal with new data without the aid of such frames, but it is also true that, when we use the same frames as our friends and colleagues, our views on the issues are more likely to win us social acceptance among the people who matter to us.
For business people whose experience has taught them that markets and competition work to make organizations efficient and motivate people to work hard, the idea of charters and choice is a no-brainer. They don’t think of themselves as employing a metaphor. When the data show that some charters outperform regular public schools and others perform less well than regular public schools and there is no evidence of net gain at scale, their response is not, “well, that didn’t work, so let’s abandon charters and choice as a strategy for improving student achievement.” It is so obvious to them from their experience in the business world that charters and choice must work that they assume that the fault must be in the implementation of the idea, not in its application to public education. More data are not likely to change this conviction. The efficacy of their solution is simply obvious to them and beyond argument.
It is no less obvious, by the way, to most teachers and parents that smaller class sizes are among the most effective of all education reforms, notwithstanding overwhelming evidence that many of the top performing countries have larger class sizes than are common in the United States, and researchers have consistently found that reducing class size is among the least effective and most costly reforms available to education reformers, except for students in the early grades. This, too, is in part the result of the application of metaphor. It is just plain obvious to us as individuals that is would be harder for us to teach 35 students than 20 and harder to teach 20 than a seminar of 12. Why is this a metaphor? Because the “egg crate” school in which each teacher works alone with a group of students is not the only way to organize a school and standing in front of the student lecturing is not the only way to organize a classroom. But, once the problem is framed this way, the evidence is irrelevant.
Then there is the problem of “tainted” evidence. There is no point, I am told, in informing American educators about the strategies that some Asian countries have used to build some of the most successful education systems in the world, because these countries have an authoritarian system of governance. I understand the impulse. But the fact remains that, on the evidence, many of the policies and practices these countries use work better than our policies and practices and will do so whether used in countries with authoritarian governments or fully democratic ones. Should we ignore this evidence just because it comes from countries with different systems of governance? Isn’t that a bit like cutting off our nose to spite our face?
More broadly, I am urged to focus our international benchmarking work on countries that look very much like the United States, or to give up our international benchmarking work altogether and focus only on identifying the best American policies and practices because Americans will never be interested in the policies and practices of other countries. Massachusetts has been leading the rest of the United States in student achievement for many years. But I am told by Dave Driscoll, Commissioner of Education in that state when it leaped to the front of the pack, that for years thereafter, no one came to Massachusetts to see how they had done it. And, to my knowledge, no comprehensive study of the strategies Massachusetts used to get to the top of the United States league tables has yet been done.
The most important reason that evidence plays only a modest role in education decision-making is that the passions and interests of human beings are involved. Education is an enormous industry. The policy and practice decisions we make will have big economic and social effects—both favorable and unfavorable—on different groups in this industry. You might have your evidence about effectiveness, but if it hurts me, my colleagues, my kids, my school or my community, you will have a fight on your hands.
The single most important challenge we face in education policy is how to build effective education systems at the scale of a state or a nation. But what I just told you is that Americans are not clamoring for evidence as to what strategies work best to build effective education systems at the scale of a nation or state. Nor have they given any indication that they would use such evidence to build more effective education systems if they had it.
Among the many reasons for that, I believe, is that our researchers most highly value research methods that are not well adapted for the study of large-scale education systems. There are a great variety of education research methods that are in use. But the dominant model is one that is designed to isolate and measure the effects of a single defined intervention such as, for example, a particular method of teaching reading, delivering professional development or financing schools. But there is every reason to believe that the superior results we see in the top performing countries are the results of the way many particular features of policies and practices are combined in their systems. The design of the whole system is the most important variable. The way the various components of the system are combined is the crucial element. The underlying principles driving the whole design are very important and the fidelity of each element of the design to those principles is no less important. The most highly regarded conventional research methods can’t tell us much about how to design effective systems on the scale of states, provinces and nations, in the sense of producing statements of the form of “A produces B produces C produces the result you are looking for.”
Worse still, the dominant model of education research is designed to produce a statement of the form: If you faithfully implement in detail intervention X, you will be likely to get results Y, under conditions Z. But, when it comes to entire education systems, no nation, state or province is interested in faithfully copying the education system of another country, province or state. That is true for many reasons, including inevitable differences in context, such as goals, values, politics, opportunities and so on. But, quite apart from that, it always makes more sense to policymakers and practitioners to combine the best ideas from many successful places than to copy only one. No one place does everything perfectly, and, even if there were such a place, no other place has the same context or culture as that of the original system. For both these reasons, mixing and matching makes much more sense that copying anyone. But, if you aren’t faithfully implementing some well described and researched intervention, researchers can’t predict what will happen with any great confidence.
The question of course is, what constitutes evidence that particular combinations of specified policies and practices— implemented at the scale of a nation, state or province and used to create comprehensive and coherent systems—are particularly effective? There is no agreement on the answer to that question in the research community.
So there we have it. Most of us, including strong advocates of evidence-based decision making, reject sound evidence if it does not confirm the limited set of frames we use to understand the world. We (including some of the best education researchers in the world) reject evidence that comes from places that are home to “them” and not “us.” And we cannot agree on what constitutes evidence when it comes to the most important question we face: how to build effective education systems at the scale of a nation, state or province.
So I expect that most of us will continue to exhort others to use evidence-based policies and practices and will, at the same time, continue to ignore the evidence when it conflicts with our frames or can’t be used to construct the systems we need.
I am not making an argument against research. What distinguishes humans from the other animals is our reasoning power. Science and the scientific method are perhaps the highest expressions of that reasoning power. I could not possibly be a stronger advocate of the use of science to address the biggest problems we face. But the idea of legislating the use of evidence in education decision-making is just a little too facile for me.
I am making an argument for modesty. We would be better off if we were realistic about the strengths and weaknesses of the research methodologies available to us for addressing the most important problems in education. We would be better off if we were just a little less arrogant when we take others to task for not following where we think the evidence leads. We would be better off if we recognized our own vulnerability when we ask whether we, ourselves, always follow where the evidence leads.
