Knowing what students know

Issues in Science and Technology, Winter 2002/2003 by Pellegrino, James W

Recent advances in the cognitive and measurement sciences should be the foundation for developing a new system of student assessment.

Many people are simply puzzled by the heavy emphasis on standardized testing of students and eager to find out exactly what is gained by such activity. The title of the National Research Council study that I co-chaired states the goal directly: Knowing What Students Know. The concerns about student assessment are quite well known: misalignment of high-stakes accountability tests and local curricular and instructional practices, narrowing of instruction by teaching to tests with restricted performance outcomes, the frequent failure of assessments to provide timely and instructionally useful and/or policy-relevant information, and the failure to make full use of classroom assessments to enhance instruction and learning.

The goal of our committee was not to review all the alleged shortcomings of past or current tests but to take the opportunity to rethink assessment. During the past 30 years, there have been major advances in the cognitive sciences that help us to understand the ways that people learn, which in turn help identify more precisely what aspects of student achievement we should be trying to assess. During the same period, there has been equally rapid progress in the development of measurement techniques and information technology that enhance our ability to collect and interpret complex data and evidence. The committee brought together experts in measurement, cognitive and developmental psychology, math and science education, educational technology, neuroscience, and education policy to determine how these recent developments can be applied to assessment. The committee's stated mission was to establish a theoretical foundation for the design and development of new kinds of assessments that will help students learn and succeed in school by making as clear as possible to them, their teachers, and other education stakeholders the nature of their accomplishments and the progress of their learning. Most assessments now in use fail to meet this objective.

We want assessment to be a facilitator of higher levels of student achievement. This will require a departure from current practice that must be guided by further research as well as policy changes. What we've learned from research enables and compels us to do a better job with assessment.

What is assessment?

Assessment is a process of gathering information for the purpose of making judgments about a current state of affairs. In educational assessment, the information collected is designed to help teachers, administrators, policymakers, and the public infer what students know and how well they know it, presumably for the purpose of enhancing future outcomes. Part of the confusion that I mentioned above stems from the fact that some of these outcomes are more immediate, such as the use of assessment in the classroom to improve learning, and others are more delayed, such as the use of assessment for program evaluation.

This means that in looking at any assessment, we have to keep in mind issues of context and purpose. Sometimes we're looking for insight into the state of affairs in the classroom, at other times emphasis is on the school system. The particular focus could be to assist learning, measure individual achievement, or evaluate programs. And here's the rub: One size does not fit all. By and large, this has been overlooked in the United States. As a result, we have failed to see that changes in the context and purpose of an assessment require a shift in priorities, introduce different constraints, and lead to varying tradeoffs. When we try to design an all-purpose assessment, what we get is something that doesn't adequately meet any specific purpose.

Any assessment must meld three key components: cognition, which is a model of how students represent knowledge and develop competence in the domain; observations, which are tasks or situations that allow one to observe students' performance; and interpretation, which is a method for making sense of the data relative to our cognitive model. Much of what we've been doing in assessment has been based on impoverished models of cognition, which has led us to highly limited modes of observation that can yield only extremely limited interpretations of what students know.

It does little good to improve only part of this assessment triangle. Sophisticated statistical techniques used with restricted models of learning or restricted cognitive tasks will produce limited information about student competence. Assessments based on a complex and detailed understanding of how students learn will not yield all the information they otherwise might if the statistical tools available to interpret the data, or the data themselves, are not sufficient for the task.

We should move away from the simplistic notion that a test is a test is a test. We have to make certain that any given assessment is designed for its specific purpose and that within that context all three legs of the assessment triangle are strong.