Scientific Research in Education/Evidence Matters: Randomized Trials in Education Research

Academe, Nov/Dec 2004 by Lincoln, Yvonna S

Scientific Research in Education Richard J. Shavelson and Lisa Towne, eds. Washington, D.C.: National Research Council, National Academy Press, 2002

Evidence Matters: Randomized Trials in Education Research Frederick Mosteller and Robert Boruch, eds. Washington, D.C.: Brookings Institution Press, 2002

For nearly three decades, the research community in the social sciences has engaged in arguments and conversations regarding the proper roles and rules of academic research. It has seemed clear that the major theoretical challenges emanating from Europe and North America, as well as the rising tide of new indigenous and postcolonial voices, would provide new, emerging models for reconsidering the primacy of positivist and postpositivist research in the United States and elsewhere. In part, these conversations have sought ways in which positivist, or conventional experimental, research could coexist with postpositivist methodologies such as phenomenological work and the work of critical theorists. Together, these three approaches might simultaneously contribute to a more precise understanding of social life and provide power, robustness, and subtlety to social policy decisions. In 1977, anthropologist Ray Rist claimed that a certain methodological detente had begun to emerge, along with a sophisticated discourse regarding what the various models of research might reasonably deliver in terms of knowledge and understanding.

These conversations and controversies revolved about the extended and extensive critiques of experimentalism's insistence on being the single best model for achieving reliable, valid, and so-called objective knowledge about social life and the social world. Experimentalism, the best known form of conventional "scientific method," is often based on random assignments of subjects to treatments, programs, or experiments. Proponents of alternative epistemologies challenged its claim to primacy and hegemony, highlighting philosophers' rejection of the possibility of gennine objectivity and the existence of multiple, sometimes conflicting, epistemologies. A sense of rapprochement had begun to settle within the social sciences, as scientists and researchers opened a dialogue around the contributions that several, rather than singular, models might make to comprehending complexity in various social and education contexts. Détente seemed achievable. With the publication of Scientific Research in Education, the 2002 report of the National Research Council's Committee on Scientific Principles for Educational Research, and Evidence Matters: Randomised Trials in Education Research, by Frederick Mosteller and Robert Boruch, it seems as though the conversation is J once again closing down.

These two new books, in conjunction with provisions of the Bush presidency's national education initiative, the No Child Left Behind Act, appear to foreclose the possibility that the federal government will support research and evaluation projects that propose multiple views of knowledge and multiple research strategies to obtain that knowledge. The No Child Left Behind Act requires "rigorous testing" of its own effects, primarily through randomized field trials, or experiments in which participants, in a natural, everyday context such as a school, are randomly assigned to receive different treatments or programs.

The National Research Council committee's aim is to delineate what is science and what is not science, and one of its fundamental assumptions is that the principles it proposes in Scientific Research in Education "provide guidance for what constitutes rigorous scientific research," largely by "identify[ing] a set of principles that apply to physical and social science research and to science-based education research," with the purpose of helping "define the domain of scientific research . . . roughly delineating what is in the domain and what is not." This would be less troublesome did the committee not "reject the postmodernist school of thought when it posits that social science research can never generate objective or trustworthy knowledge," thereby discounting most of the new theoretical critiques of Western epistemologies proposed in the past quarter century. It would also be less troubling did the committee not take several examples of contemporary scholarship and theoretical and empirical work and assign them to the realm of non-science or "unscientific" work.

The argument is far broader than education. The committee adamantly insists that many forms of scholarship and inquiry do not fit the committee's criteria and are thus "not science," when in fact most of the criteria fit comfortably with multiple forms of systematic, disciplined inquiry. For the social sciences and humanities (for example, history), the committee's limited definition of science has enormous implications, virtually eliminating scholars other than those deemed "scientific" from participating in inquiry and evaluation of social and educational programs, curriculum development efforts, or policy arenas, particularly federally sponsored or funded research efforts. The general tone of Scientific Research in Education is authoritative and authoritarian; readers are left wondering why, in light of the major theoretical critiques of positivist and postpositivist science of the past decades, the volume appears to recommit to what sociologist Anthony Giddens has called "high modernism," and to return to the now-discredited fundamental philosophies and scientific practices of the modernist era.