Artifact and Artifice in Education Policy Analysis: It's Not All in the Data

School Administrator, May, 1996 by Richard M. Jaeger, John A. Hattie

Regardless of political persuasion, those who use statistical results in support of education policy arguments are on distinctly different ground than those who use statistics in educational research.

In the latter case, statistics are used as an aid to discovery or confirmation. Hypotheses, such as "Grouping students by prior achievement level will aid teachers in delivering appropriate levels of instruction," are formulated. Statistics then are used to examine the likelihood that the converse result is plausible (e.g., "There is no difference between the appropriateness of delivered instruction when students are grouped by prior achievement level and when they are not").

Although researchers hold expectations concerning the outcome of such statistical tests (they are represented by the research hypotheses stated), the analysis is conducted to discover or confirm the data-based outcome. Furthermore, in describing the results of analysis, researchers are duty-bound to examine an array of plausible, rival interpretations, and to comment about limits on the generality of their research.

In contrast, a policy analyst who uses statistical results in support of an argument typically begins with a position of advocacy and then searches for statistical findings in support of that position. More likely than not, the findings cited are archival and are not the product of independent data collection by the analyst.

Although intelligent use of archival results and secondary analyses of data collected for purposes other than those of the researcher are to be lauded in research investigations, use of these techniques by policy analysts should raise cautionary flags in the minds of readers. Research and policy advocacy are distinctly different endeavors, and the latter gives rise to frequent and common interpretive errors and exaggerations, without the cautions typical of research reports. Some brief illustrations follow.

Beyond Analysis

Richard Jaeger, in an October 1992 Kappan article, reported on the relationship between per-student expenditures for education and average student achievement on three international assessments in mathematics.

Reacting to a policy statement by then-President Bush in which the president reportedly declared: "Dollar bills don't educate students," Jaeger showed that per-student expenditures predicted more than half the variation in average national performances in the First International Mathematics Study (overall score) and in the Second International Mathematics Study (algebra score) and just under half in the 1991 International Assessment of Educational Achievement for 13-year-olds.

Any variable that predicts half the variation in student achievement is a strong predictor. The correlation between that variable and student achievement is greater than 0.7. The data presented in support of the argument that financial investment in public schooling provides a strong return in terms of student achievement were appropriately analyzed and accurately reported.

The difficulty, however, arises when readers extend that result beyond the level of analysis used to report the finding. The relationship between per-student expenditure and average student achievement will be very different across schools, across school districts, across states within a nation, and across nations. Results for one of these levels of analysis cannot be generalized to any of the others.

An analysis conducted by Howard Wainer, in the December 1993 issue of Educational Researcher, and extended for this article illustrates this point. Wainer analyzed the relationship between per-student expenditure and states' ranks in terms of their 8th graders' average score on the mathematics test of the National Assessment of Educational Progress. His analysis included data for the 42 states that participated in the 1992 NAEP Trial State Assessment.

When he plotted states' ranks on average NAEP mathematics score against their per-student expenditure, Wainer found a modest relationship in the expected direction. We computed the correlation between Wainer's state-level per-student expenditure data and the actual average scores of the states' 8th graders on the NAEP mathematics test. We found a correlation of only 0.02, indicating essentially no relationship--in stark contrast to the strong relationship Jaeger found across nations.

The moral of this example: Do not assume that a relationship at one level of analysis will generalize to other levels.

Beyond Aggregation

In the litany of anguish over the failure of American education, an oftcited comparison is the average achievement test performances of students in the United States and Japan. Gross comparisons of average test performance confirm that Japanese students in virtually any grade and subject outscore their U.S. counterparts.

However, the story should not end with gross comparisons since overall averages often mask important differences within populations. And worse, they lead to faulty interpretations.