relationship between department rank and college rank in engineering graduate program rankings conducted by U.S. News and World Report, The

Journal of Engineering Education, Jan 2003 by Vojak, Bruce A, Price, Raymond L, Carnahan, James V

ABSTRACT

College rankings conducted by various popular magazines have generated both considerable interest and controversy. In this work, we present statistical analyses of thirteen years of U.S. News and World Report graduate program "reputation" rankings for engineering colleges and their constitutive departments, using them to reveal the relationship between department rank and college rank. Two important trends are substantiated in this study. First, we confirm statistically that some colleges with a relatively small number of top-five ranked departments place higher in the college rankings than some colleges with a significant number of top-five ranked departments. Second, we observe that college rank is much more closely related to department rank for some disciplines than others, providing additional resolution beyond our earlier work

I. INTRODUCTION

College rankings conducted by various popular magazines have generated both considerable interest and controversy over the past decade. However, while certain groups of prospective students increasingly rely on such rankings [8], a number of academics openly question their validity. Many express concern with the magazines' choices of measures and quantitative formulas used to obtain the "overall" rank for a college [1,3,5-7,12], as well as with their peers' choices of data supplied to the magazine conducting the ranking [4,11,13,14].

In the specific case of the U.S. News and World Report rankings, programs at both the college and department levels are ranked. For the purposes of this work, we use the following terminology: "departments" are the specialty units in various "disciplines" (e.g., the disciplines of electrical engineering and mechanical engineering), "colleges" are the broader units (focusing on engineering in this case) that are comprised of multiple departments, and "schools" are the broadest unit of activity comprising other colleges in addition to a college of engineering. Note that when we identify the institution from which a college or department comes, we refer to this larger institution as the "school."

For the rankings of engineering colleges by U.S. News and World Report, the "overall" rank for a college currently is calculated by using a quantitative formula that incorporates the following measures and weighting [15].

Reputation (40%)-Measured by separate surveys of both academics and corporate recruiters.

Student Selectivity (10%)-Measured by GRE quantitative and analytic scores, as well as by the proportion of applicants accepted.

Faculty Resources (25%)-Measured by student-to-faculty ratios, proportion of faculty in the National Academy of Engineering and with doctorates, and the number of Ph.D. degrees granted in the prior year.

Research Activity (25%)-Measured by total research expenditures and research dollars per faculty member.

In recent years, US. News and World Report assessed 185 engineering colleges in its annual survey [15].

Note that this "overall" rank differs from the "reputation" rank by academics. The "reputation" rank by academics is based on a survey of deans, program directors, and senior faculty who are asked to judge the overall academic quality of engineering colleges on a scale of one ("marginal") to five ("distinguished") [15].

Further, individual departments within several disciplines are also ranked. These department rankings are conducted in a manner similar to the reputation survey conducted at the college level. Deans, program directors, and senior faculty are asked to nominate up to ten top schools in each discipline, with the magazine publishing the list of schools with the highest number of nominations in each discipline [15].

It is important to note that in all cases of "reputation" ranking, the academic leaders responding to the survey use their own methods, which could be heuristic formulas, to select top-ranked colleges and departments, based on personal perceptions of relevant criteria.

The present work seeks to circumvent the confounding nature of the "overall" rank by limiting the study of college rankings to the "reputation" rankings by academics, rather than using the "overall," formula-driven rankings. By doing so, we can eliminate data value and formula bias [10] and reflect back to academics their perceptions regarding the quality of various programs, not those of the formulas chosen by a non-academic journalist.

In this work, we present statistical analyses of thirteen years of U.S. News and World Report graduate program "reputation" rankings for engineering colleges and departments, using them to reveal the relationship between department rank and college rank. In Section II, we explore the relationship between the number of top-five department rankings and college ranking, and in Section III, we address the question of whether college rank is more closely related to department rankings for certain disciplines than for others.

II. TOP-FIVE ANALYSIS

It has been noted that either the ordinary or weighted sum of departmental "reputation" rankings may not necessarily be consistent with the "overall," formula-driven ranking of the colleges they comprise [2]. That is to say that some colleges with a smaller number of top-five reputation departments rank higher in the "overall" ranking formula than other colleges with a larger number of topfive reputation departments. This is shown graphically in Figure 1 for the number of appearances in the top-five department lists for each of the top ten colleges of engineering (based on "overall" rank) [16]. Rankings for eleven disciplines were provided in the published data, shown here for 2001.