Classroom research and cargo cults

Policy Review, Oct/Nov 2002 by Hirsch, E D Jr

"We really ought to look into theories that don't work, and science that isn't science. I think the educational ... studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they've arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head for headphones and bars of bamboo sticking out like antennas - he's the controller - and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land."

- Richard P. Feynman, "Cargo Cult Science," Surely You're joking, Mr Feynman!: Adventures of a Curious Character (Norton, 19 8 5).

AFTER MANY YEARS of educational research, it is disconcerting - and also deeply significant - that we have little dependable research guidance for school policy. We have useful statistics in the form of test scores that indicate the achievement level of children, schools, and distrios. But we do not have causal analyses of these data that could reliably lead to significant improvement. Richard Feynman, in his comment on "cargo cult science," identifies part of the reason for this shortcoming that while educational research sometimes adopts the outward form of science, it does not burrow to its essence. For Feynman, the essence of good science is doing whatever is necessary to get to reliable and usable knowledge - a goal not necessarily achieved by merely following the external forms of a "method."

The statistical methods of educational research have become highly sophisticated. But the quality of the statistical analysis is much higher than its practical utility. Despite the high claims being made for statistical techniques like regression analysis, or experimental techniques like random assignment of students into experimental and control groups, classroombased research (as contrasted with laboratory research) has not been able to rid itself of uncontrolled influences called "noise" that have made it impossible to tease out the relative contributions of the various factors that have led to "statistically significant" results. This is a chief reason for the unreliability and fruitlessness of current classroom research. An uncertainty principle subsists at its heart. As a consequence, every partisan in the education wars is able to utter the words "research has shown" in support of almost any position. Thus "research" is invoked as a rhetorical weapon - its main current use.

In this essay I shall outline some fundamental reasons why educational research has not provided dependable guidance for policy, and suggest how to repair what it lacks. On a positive note, there already exists some reliable research on which educational policy could and should be based, found mainly (though not exclusively) in cognitive psychology. In the end, both naturalistic research and laboratory research in education have a duty to accompany their findings with plausible accounts of their actual implications for policy - as regards both the relative cost of the policy in money and time and the relative gain that may be expected from it in comparison with rival policies. Including this neglected dimension might wonderfully concentrate the research mind, and lead to better science in the high sense defined by Feynman.

A tale of two studies

THE NOVEMBER 2001 issue of Scientific American includes an article called "Does Class Size Matter?" about the policy consequences of research into the beneficial effects of smaller class size. The centerpiece of the article is the famous multimillion dollar STAR (Student/Teacher Achievement Ratio) study - considered to be a methodological model for educational research - which showed with exemplary technique that reducing class size will enhance equity and achievement in early grades.

But when California legislators dutifully spent $5 billion to reduce class size in early grades, the predicted significant effect did not result. Educational researchers, including the authors of the Scientific American article itself, complained that the California policy was implemented with "too little forethought and insight." Presumably this complaint implies that there are many factors that affect educational outcomes, and that we should not rely on a single one like class size. This after-the-fact criticism is valid. But if the California legislators had searched for useful "insight" in the STAR research they would have been disappointed. "Forethought and insight" cannot compensate for the deeper problem that the process of generalizing directly from classroom research is inherently unreliable.