Not-so-common "common" knowledge

School Science and Mathematics, Jan 2001 by Lederman, Norman G, Niess, Margaret L

In our reflective and scholarly moments we would like to believe that mathematics and science educators are careful readers of the literature who are open-minded and willing to learn. We openly embrace the scholarly tradition of open peer review and the idea that future scholarship benefits and builds upon the extant literature base. Was it a mathematics educator who said something about being "On the Shoulders of Giants?" Unfortunately, in our weaker moments, we forget that the data indicate that the aforementioned intentions and ambitions are not always realized.

This editorial focuses on research design and statistical analysis of data. The ideas we present are not new and have been described numerous times in previous literature. Unfortunately, researchers in mathematics and science education sometimes behave as if they have never read the literature or couldn't care less what is said. Perhaps a reiteration of several, but certainly not all, problems relative to design and data analysis will push us over the hump. Of course, we know we are preaching to the choir, because the errors discussed in this editorial surely never occur in the manuscripts published in School Science and Mathematics.

Unit of Data Analysis

A typical educational research design involves the comparison of the relative effectiveness of one or more instructional or curriculum approaches. The realities of field-based research necessitates that intact classrooms be sampled for research investigations, with these classes then randomly assigned to treatments of interest. After all, the researcher is rarely able to randomly select students for placement in groups that are subsequently randomly assigned to certain treatments. The unfortunate problem manifesting itself in data analyses is the researcher's insistence on using the student as the unit of analysis. In reality, the class (or intact group) is the correct unit of analysis, for it is the class or intact unit that was sampled, not the students. The fact that classes were then randomly assigned is a laudable approach, but is irrelevant to sample selection and the determination of the correct statistical unit of analysis. So for the aforementioned scenario, the value of "n" should be the number of classrooms, not the number of students.

This is of critical importance, because with an increasing sample size, the magnitude of a statistic needed to achieve statistical significance is much smaller. In concrete terms, the use of the student instead of the class as the unit of analysis often results in a Type I error (i.e., significant differences are claimed when they should not be claimed). The problem is not unique to education research, as it has been documented in the ecology literature under the label "pseudoreplication." Instead of intact classes, intact field plots or communities are the object of interest, and it is incorrect to use individual plants or animals (analogously, students) as units of analysis.

The problem can be described from two different perspectives. Theoretically, what has been sampled is the correct unit of analysis. The sampling unit determines the unit of statistical analysis. This point is critically important, because it is the population of sampling units to which statistical results can be generalized, not subgroups (i.e., students) of the sampling unit. Given that the class is typically sampled, the number of classes determines the unit of analysis. Whether the researcher is aware of it or not, when the student is used as the unit of analysis in studies when classes/intact groups are clearly sampled, the theoretical claim is that the distribution of students on the outcome variable of interest is the same as the distribution of classes on this same outcome variable. Some individuals may miss this point when the teacher is the same across classrooms, because they fallaciously assume that classes do not behave differently as groups if they have the same teacher. Those of us who have taught know how ludicrous this assumption typically is and reject it outright. So the second and more concrete perspective from which to view the problem of determining the unit of analysis is the recognition that classrooms function as groups in ways not represented by an analysis of individual students. Unfortunately, statistical analysis is more powerful with higher sample sizes, so there is an unconscious (or conscious) desire to use the higher number as the unit of analysis.

One way around this dilemma is to include the class as an independent variable in an effort to determine if the class is a significant variable. If it is not, it can be argued that the distribution of students is no different than the class and, therefore, there is no problem using the student as the unit of statistical analysis. Another alternative is to resist the tendency to inappropriately use parametric statistics in lieu of nonparametric statistics. The latter would remove any reliance on a theoretical population distribution. These solutions are rarely noted in the published literature or within submitted manuscripts we review for SSM or other journals in science and mathematics education. The lack of attention to use of the appropriate unit of analysis continues despite numerous discussions of the topic in the literature. Why does this problem persist in the literature? Your guess is as good as ours.