A meta-analysis of the Rossell and Baker review of bilingual education research

Bilingual Research Journal, Spring 1997 by Jay P Greene

ABSTRACT

In 1996, Christine Rossell and Keith Baker conducted a review of the literature on the effectiveness of bilingual education and concluded that the majority of 75 methodologically acceptable studies showed that bilingual education was not beneficial. This study re-examines their literature review to verify the Rossell and Baker list of methodologically acceptable studies. After identifying only II studies that actually meet the standards for being methodologically acceptable, this study aggregates the results of those studies by a technique known as meta-analysis. The conclusion of the meta-analysis is that the use of at least some native language in the instruction of limited English proficient children has moderate beneficial effects on those children relative to their being taught only in English.

During the debate over Proposition 227 in California that sought to eliminate the use of native language in the instruction of children with limited English proficiency (LEP), competing claims were made about what the research in the area concluded. Christine Rossell, for example, argued that the review of the literature she conducted with Keith Baker suggested that children learn English best when they are taught in English (Rossell & Baker,1996). Kenji Hakuta, on the other hand, argued that the review of the literature he conducted as part of the National Research Council report on bilingual education, suggested that native language approaches are indeed beneficial for children learning English (National Research Council,1997). Bewildered by these conflicting claims, the media and electorate in California generally paid little attention to researchers and Proposition 227 was passed into law.

The fact that summaries of the literature on bilingual education can be used to support diametrically opposite conclusions suggests that interpretations of research findings can be ambiguous or inconsistent. One technique to reduce the ambiguity and inconsistency in reviews of research literature is metaanalysis, a systematic and statistical aggregation of research findings. While meta-analysis does not eliminate subjective factors in interpretation, it does tend to make the assumptions of interpretation more explicit and the conclusions more rigorous. The meta-analysis reported in this article consists of a reexamination ofthe studies reviewed by Rossell and Baker ( 1996). The conclusion of this meta-analysis is that the use of at least some native language in the instruction of LEP children tends to produce moderate improvements in standardized test scores taken in English.

REVIEWING THE ROSSELL AND BAKER REVIEW

In their 1996 review ofthe literature, Rossell and Baker identified 75 studies that they determined were "methodologically acceptable."1 Studies that were determined to be methodologically acceptable had to: (a) compare students in a bilingual program to a control group of similar students; (b) statistically control for differences between the treatment and control groups or assignment to treatment and control groups had be to done at random; (c) base results on standardized test scores in English; and (d) determine differences between the scores of treatment and control groups by applying appropriate statistical tests.

These requirements for selecting methodologically acceptable studies seem like a reasonable start, but some of the items need clarification. For example, what constitutes a bilingual program and a comparable control group? What constitutes sufficient statistical control for differences between treatment and control groups?

The reanalysis of the Rossell and Baker literature review presented here adds one additional requirement and more clearly defines some of the other requirements to determine whether the 75 studies identified by Rossell and Baker are, in fact, methodologically acceptable. The additional requirement is that studies had to measure the effects of bilingual programs after at least one academic year of participation in a bilingual program. Test results after a few weeks of participation in a program should not be used to assess the effects of that program. This additional requirement only causes two studies to be excluded from the meta-analysis, one that measured outcomes after seven weeks in a bilingual program (Barclay, 1969) and another that measured outcomes after ten weeks in a bilingual program (Layden, 1972).

The requirements for a study to be considered methodologically acceptable were also clarified for this reanalysis in a few ways. First, bilingual programs were defined broadly as those programs in which LEP students were taught at least some of the time in their native language. Rossell and Baker subdivided bilingual programs into various program categories, such as transitional bilingual education (TBE), English as a second language (ESL), and maintenance bilingual education (MBE). The difficulty with these subdivisions is that program labels are notoriously unreliable descriptions of the content of the approach employed. What is called TBE is some places might be called ESL in others. The descriptions of the programs in the studies were often inadequate for drawing finer distinctions. Focusing on whether native language was part of the approach, however, is much easier to detect in each study and therefore is more likely to be a more reliable label for the programs. Besides, the policyrelevant question raised by Proposition 227 and most policy discussions is whether native language techniques in general are beneficial, not whether TBE is better than ESL. That is, Proposition 227 did not call for the abolition of a particular approach to bilingual education, it called for an end to native language instruction. Whether the literature supports that policy is an important question to address.