Student interpretation of language in student teaching evaluations

College Student Journal, Sept, 2001 by George R. Barnes, Inessa Levi, Patricia B. Cerrito

The purpose of this paper is to examine the meanings of student responses on student teaching evaluations. Approximately 100 students were randomly surveyed concerning their precise understanding of the wording in the teaching evaluation forms by slightly translating the phrasing from a school environment to a work environment. Students were asked to interpret just how often an act needs to occur if it occurs "frequently" or "generally." There is considerable variability in the student interpretation of the concepts, some of the variability associated with gender. In order to vaidate student teaching evaluations, the language must be investigated to discover in detail just what students are intending.

Introduction

Student teaching evaluations use variable language in soliciting student opinions concerning the quality of teaching in courses. Although some studies show that the evaluations present reliable and valid means of assessment of quality of teaching (Pike, 1999; Marsh and Roche, 2000; Griffin and Pool, 1998) other studies point out that teaching evaluations may be influenced, both positively and negatively, by factors unrelated to quality teaching (Peters, 1994; Simpson, 1995; Buck, 1998; Wilson, 1998; Sheehan and DuPrey, 1999). Still other papers express concern that the reliance on student teaching evaluations has inflated grades and reduced the quality of teaching (Martinson, 2000; Neath, 1996; Moore and Trahan, 1998). For example, as several studies have shown (Redding, 1998) there is a strong negative correlation between the amount of homework assigned in class and teaching evaluation scores. When correlations exist between input variables and outcomes, it is possible to influence the outcomes by changing the input variables. Thus an easy strategy to improve student teaching evaluations is to reduce the amount of assigned homework to meet students' low expectations of time spent on a course outside of the classroom. Some factors not related to teaching that influence student teaching evaluations cannot be changed. For example, some studies show that female instructors get lower teaching evaluations in traditionally male-dominated disciplines (Anderson and Miller, 1997; Bachen, McLoughlin, and Garcia, 1999; Centra and Gaubatz, 2000).

In this paper we examined the language in the evaluation questions. A typical evaluation question is formed using fuzzy expressions that have a variety of meanings. For example, the question "To what extent did the instructor help you learn the course material?" addresses students' expectations of their performance as well as the instructors' performance. Corresponding multiple choice answers: "Very little, a little, somewhat, a lot, a great deal" reflect students' personal interpretations of these concepts. To determine if there is variability in interpretation of these types of concepts, a survey of students was conducted at the University of Louisville. The exact language used in the student teaching evaluations in the College of Arts and Sciences is given in Appendix A. Because the language itself was under examination, slight changes in the wording were made in the survey; this made it possible to focus on the language itself instead of concentrating on any one course.

Validity

The survey was conducted in the spring, 2000 semester. Approximately 115 students consented to complete the survey. The survey was examined and approved by the University's IRB. The survey was administered by students in an introductory statistical data analysis course offered in the Department of Mathematics. Surveys were handed out at different times and locations on campus to achieve a reasonable representation of the student body. Student participation in survey administration was voluntary but provided credit toward the participation requirement of the course (http://www.louisville.edu/~i0levi01/560/5 60home.html.spring.2000). Graduate students taking the course are required to perform above the level of undergraduates. They were assigned the task of supervising the survey process. In this supervisory capacity the graduate students assigned locations and times to the undergraduates for collecting the data, visited them at the times to insure that proper procedures were used, and checked that data entry from paper surveys to electronic format was done correctly.

To make the survey more hypothetical, questions were altered somewhat to represent a survey of an employee concerning his/her relationship with an employer. Therefore "supervisor" was substituted for "instructor". Employee performance is evaluated; customer opinion is solicited. Student teaching evaluations, like customer opinions, are supposed to be used to improve the product. In fact, they are used more to define promotion, tenure, and merit salary increases. While a customer is not usually fired, nor will a supervisor make a decision unpopular with customers, employees do get fired, and managers do make decisions that may not be popular with employees. Therefore, the "customer" model of student-instructor relationships is not complete. Rather, a student-instructor relationship closely resembles an employee-employer relationship in the university environment since an instructor provides a limited amount of training and instruction with the assumption that the student will perform most of the coursework individually (or in a collaborative setting). In addition, the instructor evaluates the performance of the student. An employment supervisor performs similar tasks of evaluation and training.