Book review: The tyranny of educational testing and the ethics of responsibility

Journal of Literacy Research, Sep 1999 by Paris, Scott G, Paris, Alison H, Carpenter, Robert D

The central issue underlying Parts I, II, and III of the book, the first six chapters, concerns the validity of psychometric assessments of reading. The chapters present, in varying places, historical approaches to testing that mix accuracy, criticism, and indignation in a "positioned review" This review of test development and the uses of educational testing for policy making is often one-sided, the historical referent points are selective, and the stance ranges from unsympathetic toward psychology to outright antagonism toward psychometric test development. However, the authors are candid about their biases, and readers familiar with other works of Murphy, Shannon, Johnston, and Hansen will recognize the consistency of their own views over time and the mutual integrity of their advocacy position. The biases may be justified, though, from both scholarly and practical perspectives, given the fact that dissident voices have rarely been heard, much less validated through scholarly forums, and their outrage is usually muffled by the overwhelming din of political calls for educational accountability through testing. There is no death of outrage among these authors, and their passion to upend the status quo of power through testing and renegotiate the control of reading education is laudable.

But what is the authors' claim about validity? They criticize traditional notions of psychometric validity - the processes and outcomes of content, criterion-related, and construct validation - because they are based on statistical similarities in categorizing data, and hence people, and ignore the personal, social, and behavioral aspects of reading that can also count as evidence. They argue, "By focusing on inferences and interpretations, the concept of test validity is more of a social process - a process that requires deliberation rather than mechanical compliance, a process that is open to contestation, a process in which all the parties need to be given room for their voices" (p. zo). We recognize that words such as "test validity" and "voices" elicit polarizing emotional reactions among reading educators, but Fragile Evidence is not just another rhetorical battle in the current"reading wars," nor is it an outdated attack on psychometric concepts of validity. The authors suggest that newer notions of validity and reliability, based on the hermeneutic tradition (e.g., Moss,1996), demand critical analyses of evidence and interpretation that are more like joint puzzle-solving or negotiated agreements than operational definitions. The collective effort to achieve coherent interpretations of the evidence, they suggest, should be guided by professional standards, and assessment should consider the knowledge base provided by professional organizations in literacy such as the National Council of Teachers of English and the International Reading Association. The nature of the evidence, of course, depends on the theories used to interpret the evidence, the prevailing beliefs of the interpreters, and the socio-cultural-historical context of the interpretations. The authors raise the following questions about theories of reading and perforce about the nature of evidence: What counts as text? What is the relationship between language and text? What is the role of text properties? What are the relative contributions of individual psychological and social collective processes? How fixed is the meaning of text?