Inadequate conclusions from an inadequate assessment: What can SAT-9 scores tell us about the impact of proposition 227 in California?

Bilingual Research Journal, Winter 2000 by Butler, Yuko Goto, Orr, Jennifer Evelyn, Gutierrez, Michele Bousquet, Hakuta, Kenji

Inadequate Conclusions from an Inadequate Assessment: What Can SAT-9 Scores Tell Us about the Impact of Proposition 227 in California?'

Abstract

Proponents of Proposition 227 in California have argued for the effectiveness of English-only instruction over bilingual programs based on the increase in SAT-9 scores in the years since its implementation. Based on analyses of SAT-9 scores from 1998 to 2000, this article argues that: scores increased for all students, not just for English-learning students; scores increased formost districts regardless of the types of programs implemented; increases could be attributed to a number of possible factors, but it is not possible to separate out the impact of Proposition 227; and SAT-9 is not an appropriate measure for assessing English-learning students.

The scores that limited English proficient (LEP) students obtained on the Stanford 9 (SAT-9) test conducted in the spring of 2000 gained considerable media attention upon their release later that summer. The attention was in large part because the results presumably enabled an evaluation of the impact of Proposition 227, the "English for the Children" ballot initiative led by Ron Unz and passed by California voters in 1998.

The advocates of Proposition 227, highlighting the increases over the last three years of LEP students' SAT-9 scores, have argued that these validate the success of English immersion programs (e.g., Amselle & Alison, 2000). The results from Oceanside Unified School District were highlighted by antibilingual education advocates as well as by its superintendent, Ken Noonan, and received front page coverage in the New York Times (2000, August 20).2

In this article, we review a number of analyses of SAT-9 scores from 1998 to 2000. The results of these analyses clearly indicate that the SAT-9 scores of LEP students do not provide the basis for a resounding claim to victory for Proposition 227. We review below six factors that need to be taken into account in evaluating the SAT-9 scores to demonstrate why this is the case. Our results indicate how inadequate and misleading it could be to use SAT-9 results in order to evaluate the impact of Proposition 227.

SAT-9 scores rose for all students, not just for LEP students

The pattern of LEP student performance of any given school district should be considered in light of statewide patterns of performance by both LEP and native English speakers. Table I contains data on statewide reading scores specific to LEP students as well as scores for all students. The table contains data for 1998, 1999, and 2000. The numbers in parentheses indicate changes in percentile rank, and the last column shows the changes from 1998 to 2000. As one can see, there are virtually identical patterns of increases for both LEP students as well as for all students with the increases being most prominent in the earlier grades (especially grades 2 and 3). The same pattern holds for scores obtained in both math and language (see tables 2 and 3). Overall, the scores for all students as well as for LEP students rose, with large increases in the early grades, and performance tapering off in the fourth grade and beyond. It is clear from the pattern of increases seen herein that they cannot be explained simply by the effect of Proposition 227, but rather, there is something much more specific to the nature of SAT-9 causing such broad patterns of improvement.

Increases in SAT-9 scores could arise from a number of possible causes

There is relatively little information describing what specific changes in SAT-9 scores have come about as a result of Proposition 227. In fact, the increases in SAT-9 scores can be attributed to any number of possible causes. During the past few years, for instance, there has been an enormous focus on school reform in California besides Proposition 227, including class size reduction, increased school accountability, and increased focus on English language development. One potential explanation for the increased performance could simply be the effect of teachers "teaching to the test." In the past year, schools and districts in California have taken the SAT-9 much more seriously and have in many instances focused on teaching students how to perform better on the test. In addition, younger children's scores are probably more likely to benefit from increased attention by teachers and school officials to the importance of the test. We also know from experience with testing policies in other states that the first several years of a testing program show increases in scores as schools become familiar with the new test.

Moreover, districts seem to vary considerably in terms of both whom they included as LEP and non-LEP students as well as in the percentages of LEP students that they tested. The guidelines for "redesignation" for fluent English proficient (FEP) status vary from district to district, and the numbers and rates of redesignation vary from year to year, even within the same district (see Table 4, which was created based on a report by Amselle and Allison, 2000, at the READ Institute).