Organizational Report Cards. - Review - book review

Administrative Science Quarterly, Dec, 2000 by Richard E. Boyatzis

William T. Gormley, Jr., and David L. Weimer. Cambridge, MA: Harvard University Press, 1999. 272 pp. $39.95.

The anxiety of waiting for your grades in high school and taking them home to show your parents has been replaced for many of us by various comparative assessments of our organizations. Professional schools are now ranked in the media.

Despite our professorial feigned or real indifference, the results affect enrollments, donors, alumni, and the mood of deans, not to mention faculty recruiting. A sudden drop in a school's rank precipitates various attempts at reform, innovation, or blame. Are these organizational report cards a good thing for our society?

In their new book, Gormley and Weimer present a critical, conceptual view of all forms of organizational report cards. They claim that a report card is "a powerful instrument because of its capacity to draw distinctions across subjects, across time, and across persons; because of its capacity to enlighten and embarrass; and because of its capacity to propel students forward" (p. 1). Organizational report cards can serve many stakeholders. They can be used to monitor performance and allow comparisons for customers, provide accountability for regulatory agencies, and insight for policy makers. They are external assessments that transform available information so as to make it more useful to these various audiences. The authors analyze other forms of organizational scorekeeping, such as the balanced scorecard, benchmarking, and public rankings of institutions. They focus on the importance of using outcomes as the key variables. The majority of the book is an in-depth discussion of validity and comprehensiveness (chap. 4), comprehensibility and relevance (chap. 5), reasonableness and functionality (chap. 6). They weave in and out of conceptual overview and technical specifics, such as the importance of considering cohort differences in longitudinal assessment of an organization's performance.

In tackling report cards as policy instruments, they show how they can reduce information asymmetry and increase accountability and public access. The authors make a strong case for report cards as better than regulation because they are less coercive and tend to provide more accurate information on non-trivial variables. Although they use many examples throughout the book, the thorny issues of health care and education are often used for exploration. Admittedly, report cards can become a political tool, but the authors argue that they can be a democratizing tool and facilitate constructive action to improve institutions and organizations. Information, they contend, can be a public good. The government or public sector, they contend, probably has a greater capability to design a fair and accurate instrument, but the private sector has a greater capability to disseminate the information engagingly. The media, they point out, is in the private sector. They make a case for private and public sector responsibilit y in creating and maintaining these report cards.

A minor criticism of their thoughtful discussion of audience demands and needs is that it takes a decidedly third-party approach. Executives and leaders of institutions are portrayed in an almost reactive mode. The greatest utility of organizational report cards may be as useful information to executives of organizations, as an aid in organizational improvement, as a gauge for measuring continuous improvement, and as a catalyst for organization development or transformation. The authors offer a discussion of functional organizational responses, such as process improvement, input reallocation, managerial focusing, and mission enhancement. They also discuss dysfunctional organizational responses as self selection, cream skimming, teaching to the test, deception, and blaming the messenger. Those of us involved in professional and higher education's emerging arena of "outcome assessment" can offer numerous examples of the intellectually stimulating and organizationally energizing impact of such assessments of how our students change and/or what they learn in our programs (Banta, 1993; Boyatzis, Cowen, and Kolb, 1995).

"Effective report cards compare organizations in terms of their performance. Measuring performance generally requires that some outcomes, or their proxies, be adjusted to take account of the difficulty of bringing them about. The task, commonly referred to as risk adjustment, ranges from the relatively simple calculation of ratios ... to the complex construction of counterfactuals" (pp. 205-206). The utility of comparative assessments of institutions rests on such risk adjustments. The obstacles to appropriate risk adjustment may be technical, such as errors arising from oversimplification in creating a unidimensional measure. The authors contend that the obstacles are more often political, such as the convenience and cost advantages of using standardized tests such as SATs and GMATs to assess academic progress. These tests are easy to administer but are subject to social class bias and are valid predictors of little other than grades (McClelland, 1973, 1994).