Cross-posted from Education Week
NCEE has been benchmarking the education systems of the top performing systems worldwide since 1989. This has prompted some reflections on the methods and substantive focus of education research, especially education research in the United States, which I will share in this and the next few blogs.
The achievements of the American education research establishment, with respect to the sophistication of method and the power of the findings, are very impressive. In every corner of the globe where we do our own research, we find people directing the most successful education reforms in the world talking about their debt to American researchers.
Yet something is amiss here. The American educational research establishment is by far the largest in the world. Yet the performance of the American education system is mediocre, and falling fast, relative to the achievements of the top performers. The whole point of public support for education research is the assumption that it is a very powerful driver, some would argue the most powerful driver, of improvement in education outcomes. What’s wrong? Is there less resistance to the findings of American education researchers outside the United States than within our own country? There is no evidence of that. Are the findings from our researchers more applicable in foreign than domestic settings? That would be bizarre.
The answer, I think, lies in the difference between the way the American education system is governed and the way the education systems in the top performing countries are governed, a subject about which I have written elsewhere. In the top performers, ministries of education at the national level and, in some countries, the state level, have virtually complete authority with respect to education policy. Not so in the United States, where the constitution greatly circumscribes the authority of the U.S. Department of Education, states’ departments of education are typically very weak, there are many other entities at the state level vested with responsibilities that, in another country, would be vested in the ministry, and local boards of education have far more authority, and often, much more staff, than their opposite numbers in the top performing countries. This is not decentralization. It is more accurately described as dispersion of authority.
The result is an education system that does not function as a system. In a properly functioning system, each of the parts and pieces is designed to work in harmony with all the other pieces. When one part adapts to changes in the environment, the related parts adapt accordingly. That is very much what we see in the top performing countries. But that is not what we see in the United States, where policies are rarely designed to support other parts and pieces and, in some cases are deliberately designed to frustrate the faithful execution of other policies, designed by people who have different policy objectives and embrace very different solutions to the issues of the day. In our system, elements of policy on the qualifications of teachers are often made by people who are at best unconnected to the people who set the standards for program of schools of education and both are different from the people who have the authority to waive the requirement for any standards for teachers and all of these are unrelated to the people who set teachers’ compensation and still others decide on most of the conditions of work. This set of facts makes it impossible to have a coherent human resources system in our schools. Each actor is at the mercy of all the others. In the top performing countries, examinations are based on a curriculum set by the state, but, in our country, the state might set the exams, but cannot do so based on the curriculum, because that is determined by the district, the school or the individual teacher. Curricula are set by people with very different values, interests and tends therefore to be a hodgepodge to which any given assessment may or may not be related. As a result, instructional systems in the United States typically lack the coherence found in the top performing countries.
The American approach to determining, through research, what works, is to identify some highly bounded problem, famously, for example, the teaching of reading, and look at contending programs, or in the medical terminology on which the research methodology is based, “treatments” that purport to address this challenge, and create, either physically or by statistical manipulation, a situation in which all variables other than the treatment are neutralized, so that the independent effect of the treatment can be measured.
Again and again, the effect sizes measured by these techniques are disappointingly small, but the search goes on. What is crucially important to grasp here is that, for forty years, the net change in performance of the American system is nearly zero, which no one has explained, and most researchers simply ignore. From my point of view, the explanation is obvious. The system itself is dysfunctional, for the reasons I have just described. Because all researched solutions live within this dysfunctional environment, one could expect little change if the system itself is not changed. Let me be a little clearer about what I mean by “dysfunctional.” Because policies are designed without reference to each other and all too often to counter one another by people who disagree with one another, it is hardly surprising that for any given actor, the incentives to do things that actually benefit students can at any given time be positive, neutral, negative or even perverse. In that environment the positive effects associated with one policy or practices are as likely as not to be cancelled out by the negative effects of another. I am speaking here, for example, about policies and practices having to do with curriculum, standards, assessment, teacher quality or school organization and management. The programs that would show the best results would not be the programs with the highest potential for success, but the programs that are best adapted to the demands of this dysfunctional system or those that are based on the most detailed and prescriptive designs, constructed that way in an effort to overcome at least some of the chaos.
And that is exactly what we see, everywhere. The most prescriptive programs and those that challenge the status quo the least are indeed the ones researchers find most effective.
But this is not the worst of it. If my colleagues and I are right, and the design of the system itself is the reason American schools are steadily falling behind the top performers, the biggest problem with our research effort is the lack of attention that our researchers pay to the design of the system itself. We are focusing with great sophistication on the constituent elements of the system, but hardly at all on what makes for effective systems. Our research has become a constant search for suboptimal solutions. When we start researching what it takes to create high performance systems, we will find that many programs now deemed to “work” will be found much less effective than policies that are not about programs but which, when joined with others, create systems in which students routinely perform at high levels.
I will return to this point in my next blog. For now, let me leave you with the thought that the real failure here is a failure of theory, not of methodology. American researchers, as I see it, have not really come to grips with the implications of the precipitous fall of the United States in the global education rankings. And, as a result, they have not really grasped the nettle, the importance of the design of education systems. They are focused on the trees, not the forest. They do not yet have a bundle of plausible hunches to explain the superiority of other nation’s systems on which theories could be developed and then tested. Until that happens, our very sophisticated research establishment will not be able to help us as much as it could. When we get some decent theories, we will see whether we need new methods to deal with them.
