Developing a psychometrically sound measure of collegiate teaching proficiency

College Student Journal, March, 2008 by Donald C. Barnes, Brian T. Engelland, Curtis F. Matherne, William C. Martin, Craig P. Orgeron, J. Kirk Ring, Gregory R. Smith, Zachary Williams

While student evaluation of teaching (SET) has become a common practice on most college campuses, research suggests that weaknesses exist in many of the common instruments employed for this assessment. This study lays the groundwork for the development of an improved psychometrically sound measure of teaching proficiency that can be used in a university setting. An inductive approach is used to explore the construct domain, resulting in the proposition that teaching proficiency is composed of two separate dimensions: teaching readiness (TR) and teaching excellence (TE). Scales for each dimension are developed and preliminary assessments are conducted for dimensionality, reliability, discriminant validity and nomological validity. Use of the scales is recommended to improve the quality of the SET process..

**********

Student evaluations of teaching (SET) are administered in almost all American colleges and universities and are probably the main source of information used for evaluating faculty teaching performance (Abrami & d'Apollonia 1997; Cave et al. 1997; Simpson & Siguaw 2000). Writing in the Chronicle of Higher Education, Wilson (1998) contends that over 2000 studies have been completed that delve into the subject of student evaluations of faculty, thus creating a vast scholarly literature from which to draw conclusions regarding the reliability and validity of student evaluations of faculty (Centra 1993).

Unfortunately, the literature suggests that many of the instruments employed for SET assessment have weaknesses, including inconsistency in domain definition, lack of correlation to student achievement, and uneven item quality (i.e., Bosshardt & Watts 2001; Engelland 2004; Young & Shaw 1999). While scholars have endeavored to develop precise measures of teaching proficiency, faculty committees have exerted pressures to add additional items to the assessment which may or may not fit the construct definition. These additions can work against objective comparisons of teaching performance across courses, colleges and universities, and result in different viewpoints as to the underlying factors that comprise the construct (Marsh 1987; Marsh & Roche 1997). While research has focused upon the identification of best practice characteristics, behaviors and traits, studies have concluded that many of the resulting SET measures are unrelated to student achievement (Cohen 1983; Dowell & Neal 1982; McCallum 1984; Tian 2000). Furthermore, a review of SET instruments used on over 300 college campuses suggests that extant scales are uneven in quality, with many containing double-barreled, unclear, and spurious items (Engelland 2004).

The objective of this study is to lay the groundwork for the development of a psychometrically sound measure of teaching proficiency that can be used in a university setting. In the following pages, we present a step-by-step commentary on our efforts to define the teaching proficiency construct, develop scale items, purify the measure, establish dimensionality and finally assess reliability and validity.

What Is Teaching Proficiency?

In order to measure anything, a precise definition of the construct of interest must be agreed upon (Venkatraman 1989). Unfortunately, the literature dealing with this construct has not coalesced on a common definition. Measures used at some institutions model teaching proficiency as a formative construct, wherein the successful performance of various categories of activities defines good teaching (MacKenzie 2003). Other assessments have modeled the construct as a reflective measure, wherein the instructor's overall teaching proficiency influences measured student perceptions. The dimensionality of the construct has been modeled with as many as nine dimensions (Marsh 1984, 1987) and as few as one dimension (Abrami & d'Apollonia 1997). All of this work has resulted in a lack of clarity in what constitutes proficiency in teaching and how it should be measured.

In order to begin to resolve these issues related to measurement, a panel of eight expert judges, representing scholars in management, marketing, political science and psychology, followed an inductive approach to construct definition. This inductive approach began by gathering copies of nine SET instruments in common use on over 300 college campuses, and evaluating the items contained in these instruments for quality. After removal of double-barreled, duplicate and similarly-worded items, 79 different items remained for subsequent analysis. Next, two judges formed the items into seven logical clusters based upon common themes. The clusters were: (1) preparedness, including items relating to how well the instructor is prepared for teaching; (2) professionalism, including items related to the instructor's knowledge, respect for students, and effectiveness in implementing course objectives; (3) evaluation, which included items relating to whether the instructor evaluates students appropriately in a timely and objective manner; (4) rapport, which included items relating to the relationship established between instructor and student; (5) enthusiasm, which included items relating to eagerness and passion which the instructor displays in teaching; (6) delivery, which included items relating to how effective the instructor is in conveying knowledge to students; and (7) excellence, which included items relating to a global assessment of the student's perception of the instructor's teaching excellence that did not address any specific teacher attributes.