Cross-posted from Education Week
Last month in his Education Week blog, Bob Slavin poured a lot of cold water over the rising interest in learning from other countries that are routinely outperforming the United States. He argues that, while we can identify interesting practices, we can never be sure that those practices are causing the student achievement we admire. We have plenty of variation right here in the United States, he says. Why not research that? Americans, he says, won’t outperform the Finns until we “start implementing more effective programs and practices, proven in research in our schools”.
I’ve long admired Bob and his work, but I take exception to this argument. American education research has long favored the experimental method, which calls for holding all variables constant (either in fact or using appropriate statistical techniques) except the variable of interest. The ideal is random assignment of subjects to alternative treatments. But the world is a messy place, and the education world is messier than most. In the real world, it is often possible to exert substantial control over the variables of interest when studying a particular aspect of reading instruction. But it is impossible to do so when studying an entire education system.
Why? Well, if we were to apply “gold plated” American research techniques to the study of national or state education systems, we would have to randomly assign national populations to national education systems and keep them isolated from each other long enough to make sure there was not contamination of ideas among the treatments and the populations. Parents in Scarsdale, New York might take exception to having their children shipped to the schools of Estonia for such purposes. Random assignment of whole national populations to national education “treatments” will never happen.
I assume that Slavin wants us to do our research here in the United States because the context in which it would be done is the American context and so it would be much easier to apply it here. But that is just the problem. The reason that other countries are doing better is because they are doing things differently. It is those differences that we need to identify and understand. By definition, we won’t find them here. We have to go there. I would argue, in fact, that the differences in practices we find will be less important that the differences in the nature of the state and national systems. And it is precisely those systems that cannot be found anywhere in the United States.
In many respects, Minnesota has more in common with Finland than with Mississippi. The one thing they don’t have in common is the performance of their students. Finland’s students outperform those of Minnesota and there is an urgent need to find out why. One nation after another is surpassing the United States as a whole in student achievement, and surpassing our best states’ performance as well. Slavin’s argument reminds me of the old joke about the drunk looking for his keys under the streetlight, though he knows he lost them near the park bench 20 feet away. “Why are you looking there?” a stranger asks. “Because that is where the light is,” the drunk replies. If we want to find out how the top-performing countries are doing it, we better look at what they are doing.
With respect to methodology, Slavin is wrong on two counts. First, as I just explained, American preferences in research methodology are singularly ill-suited to the paramount research challenge of our time: Understanding why some complex national systems of education produce better results than others. Second is his implicit assumption that, somehow, the appropriate methodologies will be applied to better effect at home than abroad. If you want to see rigorous research in action, it is hard to top the work reported on by Jim Stigler and Harold Stevenson in their classic comparison of instructional systems in Japan, China and the United States, The Learning Gap, or Ludger Woessmann’s various statistical analyses of the PISA data, or Liping Ma’s brilliant comparative study of the teaching of elementary mathematics in China and the United States, Knowing and Teaching Elementary Mathematics.
Slavin jokes that when all is said and done, we have no way of knowing whether Finnish students’ superior performance is caused by superior teachers or their consumption of smoked fish. Apart from the fact that it is no easier to determine the cause of the superior performance of Minnesota’s students than of Finnish students, for the same reasons, he is just plain wrong. Because we now have very highly regarded common measures of student performance across countries, we can now make valid comparisons of performance across those countries. Because we can collect a rich variety of background information from those countries, we can do very complex and revealing correlations among the outcome data and the data relating to other important variables, and we can see the patterns that emerge.
Studies of that kind can tell us a lot about which countries are better than others at reducing the effect of parent’s socio-economic background on student achievement and at relating that information to information about the equity of school funding in those countries and the variation in student support systems that might account for the variation in the influence of parents’ background on student performance.
But studies of that sort cannot tell us what policies were put in place to change school funding or to provide more effective student support systems, what strategies were used to build support for those strategies, how opposition was overcome and what kind of practices were put in place to support the policies. If you want to see the connections among plans, actions and results, you need researchers’ boots on the ground. For that, another kind of research is required, involving techniques from a broad range of social sciences. It would take no time at all for research like this to eliminate smoked fish from the list of possible cause of superior Finnish educational performance, and to put the finger on teacher quality. Why? Because, again and again, in study after study of the world’s best education systems, we see successful efforts to raise teacher quality followed by big improvements in student performance and we fail to observe increases in the consumption of smoked fish followed by the same result.
It is time leave the street light and find a way to put some light on the park bench.
