Spending time wisely: A model for teaching improvement through enhanced analysis of student evaluations

NACTA Journal, Sep 2003 by McCann, Laura, Burton, Michael

Abstract

Student evaluations are used for both formative and summative assessment of teachers. A method is provided to make more effective use of these student evaluations by individual teachers. In particular, improved analysis of this information can enable teachers to allocate their time more efficiently. Likert scale data from three years of evaluations in two courses were used to develop regression models to explain overall effectiveness of teaching. For a course in Environmental Policy, the coefficient for Explanations (0.59) was much larger than that for Organization (0.17), indicating that more effort should be spent on improving explanations than improving organization. For a course in Economics of Agriculture and Natural Resource Management, Enthusiasm had the largest coefficient (0.41), followed by Explanations (0.31), Pace (0.22) and Knowledge (0.16). Therefore, depending on the course, it may be necessary to allocate effort to different aspects of teaching.

Introduction

University professors are increasingly time constrained. Therefore, the time that can be spent on teaching improvement is scarce so they must efficiently allocate this time. One way to do this is to use student evaluations more effectively. Analysis of the responsiveness of overall teaching quality (measured by an effectiveness rating) to improvements in various aspects of a course can have a significant impact on the productivity of teaching effort by enabling the teacher to focus on revising the aspects where the marginal product of effort is greatest. A simple examination of the student evaluations does not allow the teacher to evaluate the strength of the relationships between various aspects of teaching, and thus does not enable targeting of teaching effort.

Student evaluations of teaching have been used for a number of purposes (Ory, 2000). In the 1960s, they were promoted by students to improve public accountability and to help students make decisions regarding course selection. In the 1970s, the primary focus was on development of faculty. In the 1980s and 1990s, teaching evaluations have been extensively used by administrators for promotion and tenure decisions. A 1978 survey of deans of liberal arts colleges found that 55% of deans always used systematic student ratings to evaluate teaching performance, by 1998 this number had risen to 88% (Seldin, 1999a). This is due, in part, to an increasingly litigious climate on campuses that requires decisions be based on objective criteria (Ory, 2000; Seldin, 1999b). Fortunately, hundreds of studies have determined that, overall, student ratings are both reliable and valid (Soldin 1997). In a meta analysis, Cohen (1981) found that the correlation between course ratings and mean student achievement was 0.47. He also found that global ratings, such as overall teaching effectiveness, correlated more highly with student learning than more specific items.

A more recent trend is the use of a wider variety of measures of teaching effectiveness in personnel decisions and a more structured and systematized process for collecting data (Seldin, 1999a). For example, self-evaluation was always used by 37% of deans in 1978 and by 59% of them in 1998, while the use of course syllabi and exams increased from 14% to 39% over this period. Teaching portfolios, another form of evidence, are increasingly being used to evaluate teaching performance. Zubizarreta (1999) indicates that a teaching portfolio contains selective evidence of current teaching accomplishment to document performance but also reflective analysis to improve teaching. Reflective analysis of student ratings and a description of how it was used to improve teaching can be one component of a teaching portfolio.

Seldin (1999a) indicated that there are more than 15,000 studies on teaching effectiveness. There is consensus on the characteristics of good teaching. In a survey article, Eble (1988, p. 21) found that "Most studies stress knowledge and organization of subject matter, skills in instruction, and personal qualities and attitudes useful to working with students." In the area of economics, a review of the literature found that certain variables, such as organization and clarity influence teaching effectiveness (Boex, 2000). However, teachers need to be able to answer the question: "What should I do to improve?" in addition to "What characteristics are associated with good teachers?" The average scores for items on student evaluations do not provide adequate information about the areas that are most important to the overall level of student satisfaction with the course, and thus, provide little guidance as to how to allocate one's teaching improvement effort.

McKeachie (1975) found that ratings can lead to improvement in teaching if 1) ratings reveal something new about the teacher's performance, 2) the teacher is motivated to improve, and 3) the teacher knows how to improve. Open ended questions as well as diagnostic questions or questions on specific teaching behavior should be included in student evaluation forms to increase their usefulness for faculty development (Seldin, 1997; Cashin, 1999).