The Mismeasurement of Educational Quality - education

School Administrator, Dec, 2000 by W. James Popham

School leaders have a duty to undo the harm being perpetrated on schools today

America's educators are being obliged to participate in a contest they can not win. The contest is called "the scoreboosting game" and its object is to have educators raise students' scores on standardized achievement tests.

The score-boosting game rests on the assumption that a set of high scores by students equals more successful instruction by educators. That assumption, however, is misguided because it clearly misapplies the information that can be gleaned from standardized tests.

Unfortunately, many educators' current efforts to succeed in this accountability-induced contest are causing serious educational harm to students. Today's evaluation of schoolwide performance, based on students' standardized test scores, must be ended. Fortunately, there are ways this brand of educational mismeasurement can be reduced.

Mischief Makers

I am not writing this commentary to be provocative or to beef up my publication list. I am doing so because the country's top educational leadership has a responsibility to do something to mitigate the harm now being perpetuated among students. Superintendents and their colleagues in decision-making positions need to halt the significant educational mischief now going on in our schools because of policymakers' misguided approaches, inspired by ignorance, for holding schools accountable for academic performance.

In what follows, I will suggest why standardized achievement tests do not yield an accurate picture of a school staff's instructional effectiveness. I'll describe a few harmful educational practices that have been spurred by educators sometimes mindless pursuit of higher test scores. Finally, I'll set forth six specific actions that the nation's educational leaders could follow to diminish or expunge this increasingly widespread misuse of standardized achievement tests.

Simply put, bad things are happening to children in many schools because of an unsound accountability strategy rooted in the wrong type of tests. The strategy was initiated by educational policymakers who mistakenly believed that students' scores on a standardized achievement test tell us how well those students have been taught. Nor do I fault the architects of these well-intentioned but misguided testing programs. State legislators and state board of education officials were attempting to help children on the basis of their flawed knowledge about educational assessment.

Rather, I fault us. That is, I fault the education professionals (myself included) who sat on their hands while the score-boosting game was born and multiplied. Our apathy has allowed this mismeasurement of educational quality to flourish and led to a situation that has placed teachers and administrators in an untenable position. Even worse are the adverse educational consequences for our students.

Measurement Misuses

A standardized achievement test is any examination that is administered and scored in a standard, predetermined manner. The five commonly encountered standardized achievement tests are the California Achievement Tests, the Comprehensive Tests of Basic Skills, the Iowa Tests of Basic Skills, the Metropolitan Achievement Tests and the Stanford Achievement Tests. But in a number of states, we also see standardized achievement tests that have been built to mesh better with a state's curricular preferences.

Today's standardized achievement tests are modeled after the Army Alpha test, developed in 1918 to help identify suitable officer candidates for America's World War I forces. The Alpha was an intelligence test measuring an Army recruit's aptitude for success in an officer training program. The Alpha's comparative assessment strategy allowed recruits to be compared with a norm group (of previous Alpha test-takers) in order to isolate each recruit's relative aptitude.

Today's standardized achievement tests follow the Alpha assessment strategy. Therefore, today's standardized achievement tests must produce a sufficient degree of score spread so that fine-grained comparisons among students can be made.

I have no quarrel with standardized achievement tests if they are used properly. Both educators and parents can be helped by learning that a child scored at the 86th percentile in mathematics but only at the 30th percentile in language arts. That's a good use of such tests. But standardized achievement tests do not tell us how well students have been taught.

Here are three reasons why that's so.

First, because the makers of these tests must market their exams on the basis of a one-size-fits-all curriculum, what is taught in many localities does not match what is tested. Some studies suggest that more than 50 percent of the content on many standardized achievement tests is not taught, or not even supposed to be taught, in some school districts.

Second, because of the tests' relentless quest for score spread, test makers dare not include many items that fail to discriminate among students, Items that many students answer correctly tend to be excised from these oft-revised tests because they do not contribute to score spread. Yet items on which students score well often turn out to deal with content that, because of its importance, teachers stressed. Thus, items covering the most important things that teachers teach will tend to be jettisoned from standardized achievement tests.