Improving a School's U.S. News and World Report Ranking

Journal of Engineering Education, Jul 2004 by Tsakalis, Konstantinos S, Palais, Joseph C

ABSTRACT

In this paper we analyze the 2002 U.S. News and World Report rankings of universities offering graduate programs in engineering. We show that a reasonably simple model, whose independent variables are derived from the intuitive causal relationships among the various performance sub-measures, can represent the data. The results can be used to quantify the choice of an efficient policy for improving a school's ranking.

Keywords: school rankings, school-evaluation metrics, U.S. News and World Report

I. INTRODUCTION

A recent paper [1] reported on the relationship between department rank and college rank in the U. S. News and World Report results. A related paper [2] correlates productivity and quality factors with the U.S. News and World Report rankings of chemical engineering departments. In this paper, we develop a control model based on the 2002 U.S. News and World Report data and provide an interpretation of the results [3].

The U. S. News and World Report has created considerable discussion and controversy with their model for school ranking. Part of the debate is attributed to the subjective components of the ranking model and a lack of validity of the factors that contribute to them. While the subjective components of the U.S. News and World Report model do take input from a large number of respondents, nonetheless, a small change in a school's scores can significantly influence the resulting ranlang. Further, the U. S. News and World Report model, like other models [4-6], incorporates a number of quantitative measures that are intuitively expected to reflect the quality of the school, such as research expenditures, studentfaculty ratio, and admission standards. Combining such diverse statistics into a single metric, like the subjective components, lacks validity as a measure of the quality of graduate education.

Regardless of one's opinion of the utility of the U.S. News and World Report rankings, they do invite considerable analysis by colleges and often with the objective of gaining insight into the factors that most influence their ranking. In this paper, we develop a control model from the U.S. News and World Report data. We emphasize that there is a conceptual difference between a control model and a data fit or predictive model. While the former attempts to relate independent (manipulated) variables to the quantities of interest, the latter simply approximates (or predicts) the value of one variable given the values of the others. Within its region of validity, such a model can provide answers to several "what-if' scenarios, but the user is responsible to ensure that the selected adjustments are feasible. Using the (usually simpler) control model instead, the predictions are less accurate but the feasibility question is much easier to assess.

In the next section, we analyze the apparent sensitivity of the U.S. News and World Report rankings with respect to the various sub-measures of performance. With emphasis on casual relationships, we create fitting models to assess the incremental effect of various improvement policies on school ranking. Our results allow us to examine the influence of the primary manipulated variables (faculty size, expenditures, number of Ph.D. students, and percentage of faculty in the National Academy of Engineering) on the school ranking and provide a means to establish a cost-efficient direction of improvement. The Ira A. Fulton School of Engineering at Arizona State University is used as an example.

II. THE U.S. NEWSAND WORLD REPORT RANKING SYSTEM

The 2002 published report ranked 185 graduate engineering schools (164 responded to the survey and 145 provided all data) based on eleven sub-measures of performance. According to their disclosure, the sub-measures of performance and the corresponding weights are given in Table 1. The correlation coefficients between each sub-measure and the total score are also included in this table.

The total U.S. News and World Report score for each school was computed as a weighted sum of all individual z-scores (normalized by variance and a logarithmic transformation). The weights were chosen after performing the survey.

Although the U.S. News and World Report's methods and rationale are well documented and provide (arguably) valuable information to potential students, they are not directly amenable to analysis. For example, the published data do not include master's students and they only include data on the schools ranked in the top 50. While recovery of the master's data was possible through a tedious manual process, a complete duplication of the results would require data from all the schools in order to compute the variance and the normalized scores. The omission of this data might be justifiable to prevent "micromanaging" of the information, which may dilute the objectivity of the ranking system. On the other hand, one can argue that such a detailed analysis of the data can actually help the schools to improve on measures that are perceived as defining the quality of education.