To catch a cheat: the pressures of accountability may encourage school personnel to doctor the results from high-stakes tests. Here's how to stop them - Research

Education Next, Winter, 2004 by Brian A. Jacob, Steven D. Levitt

THE COSTANO SCHOOL IN EAST PALO ALTO, CALIFORNIA, ACHIEVED national recognition in 2000 for overcoming enormous obstacles to achieve academic success. Its principal, Marthelia Hargrove, was designated Principal of the Year by the National Alliance of Black School Educators, and the school was identified by the nonprofit Education Trust as one of the relatively few high-poverty, high-minority schools nationwide that perform well on state exams.

Just two years later, however, the school's performance came under question when 11 former students, interviewed by the San lose Mercury News, acknowledged that they had received inappropriate help from teachers on state tests. A former Costano teacher told the paper that a school administrator had encouraged him to cheat on the exams, and a subsequent examination of the students' actual answer sheets revealed an unusually high number of erasure marks where answers had been changed from wrong to right.

Costano does not appear to be an isolated case. Similar incidents have been documented in Indiana, Maryland, New York, Texas, and Virginia, to name just a few. These scandals have aroused public concern, but there has been little hard evidence on the extent of cheating by school personnel on the type of tests required by recently enacted accountability legislation. Our research on cheating in Chicago indicates that the problem is significant enough to require attention, but not so widespread as to call into question the integrity of the nation's educators. A statistical technique that identifies cases of potential cheating within Chicago's public schools (described in detail below) indicates that, on any given exam, only 3 to 6 percent of classrooms experience instances of teachers or administrators' doctoring students' exams.

Still, with the implementation of the No Child Left Behind Act, the incentives for teachers and administrators to manipulate the results from high-stakes tests will only grow, especially as schools begin to feel the consequences of flow scores. No matter what the outcome, cheating hurts the cause of school improvement. When they are not caught, school personnel who cheat may avoid accountability and thereby also cheat their students and the public. When they are caught, these school personnel produce scandals that drain energy and resources from the real business of schools and reform. With better detection methods, like the technique described here, schools can both investigate potential abuses of the system and deter cheating in the future, both of which will increase confidence in the results from high-stakes tests.

This article describes the results of a three-year investigation into cheating by school personnel. The goals of this research were to measure the prevalence of cheating by teachers and administrators and to analyze the factors that predict cheating. Strategies for cheating can include altering students' answer sheets, giving students the answers, or obtaining copies of an exam before the test date and literally "teaching the test"--prepping students with answers to actual test questions. Using data on test scores and student records from the Chicago Public Schools, we developed a statistical algorithm to identify classrooms where cheating was suspected. This method depends on two hallmarks of potential cheating: unexpected fluctuations in students' test scores and unusual patterns of answers for students within a classroom. At the invitation of Arne Duncan, chief education officer of the Chicago Public Schools, we were given the opportunity to work with Chicago administrators to design and implement auditing and retesting procedures in 2002. The results of this retesting provided strong support for the effectiveness of our method for detecting cheating. Finally, we examined whether cheating responds to incentives, notably the introduction of a high-stakes testing regime in 1996.

Testing Procedures

Each year, Chicago students in grades 3 through 8 are required to take the Iowa Test of Basic Skills (ITBS), a national norm-referenced exam with a reading comprehension section and three separate math sections, for a total of four subtests. The sample used in our analysis included all students in grades 3-7 for the years 1993-2000. (We excluded 8th graders because our statistical method requires scores for the following year.) Our data included each student's answers on each year's test; which school and classroom each student was in; each student's previous and future test scores; and demographic variables including each student's age, sex, race, and eligibility for the federal school lunch program, a widely used proxy for family income. The data also included a wide range of school-level characteristics. However, the data did not allow us to directly link teachers to classrooms or to track a particular teacher over time. Our final data set contained roughly 20,000 students per grade per year, distributed across approximately 1,000 classrooms, for a total of more than 40,000 "classroom years" of data (with four subject tests per classroom year) and more than 700,000 "student year" observations.