Development of a Work Sampling Methodology for Behavioral Observations: Application to Teamwork

Journal of Engineering Education, Oct 2007 by Besterfield-Sacre, Mary, Shuman, Larry J, Wolfe, Harvey, Clark, Renee M, Yildirim, Pinar

ABSTRACT

Engineering programs must assess students' abilities to master "criteria 3 a-k." Skills such as teamwork, problem solving, design, and ethical understanding entail learning various processes; hence, assessing these outcomes is better accomplished by focusing on the process rather man the result Methods for observing students' performance, such as 100 percent behavioral observation, are ideal but expensive.

We extend work sampling, an economic industry-based alternative, to observe cognitive and behavioral processes. Specifically, we describe a work sampling methodology to assess students engaged in teamwork. We then determine attributes of teamwork, establish target time proportions using 100 percent observation, and statistically compare the targets to proportions obtained from work sampling intervals to determine the effective interval The robustness of work sampling is tested in four learning environments. Results indicate that sampling provides a statistically valid alternative for assessing teamwork. However, when observing design and ethical understanding processes, additional research is needed to make work sampling viable.

Keywords: behavioral observation, teamwork, work sampling

I. INTRODUCTION

The engineering education accreditation criteria adopted by ABET in 1997 introduced a continuous improvement process with a focus on assessing a minimum set of eleven outcomes related to cognitive, affective, and behavioral processes [1]. This has motivated the need for robust, in-depth measurement methods that can fully assess these outcomes. Although it has resulted in substantial improvements in engineering education assessment over the past 10 years, much more remains to be done. During the early years of the new accreditation criteria, engineering programs typically measured the 11 outcomes with indirect assessments; the two most widely used methods being focus groups and surveys [2] of program constituents, primarily students and alumni (the latter of whom also represented industry). Although such assessments were useful for obtaining students' perceptions of their education in progress and some limited (but often dated) industry feedback, the engineering education community and ABET officials quickly recognized that these methodologies were insufficient outcome measures. One result was a call for more direct measures for both the "hard" and "professional" outcomes. Included among these emerging direct measures are performance appraisals (i.e., projects, capstone design courses) and simulations, both of which are now being coupled to rubrics in order to obtain the actual measurements [3]. Specific instruments also have been developed or adopted from other educational settings to address certain outcomes. These include, but are not limited to: the Team Developer (now the Professional Developer) to measure teamwork [4], the Intercultural Development Inventory (IDI) to measure cultural sensitivity (3.h) [5,6], and the Self Directed Learning Readiness Scale (SDLRS) as a surrogate measure for life long learning (3.i) [7,8].

That said, ABET's focus on continuous improvement, even with these limited assessments, appears to be improving engineering education, as documented by a recent study commissioned by ABET and funded by the National Science Foundation (NSF) [9]. However, as engineering educators become more expert in measuring student outcomes, the next phase of assessment should shift the focus to the actual processes students use to achieve these outcomes. In doing this, engineering educators will start to address student cognition and higher dimensions of learning. That is, assessment will be concerned with measuring students' abilities to retain knowledge and skills and to transfer that knowledge to new problems, something that a few engineering educators are already observing [10-11]. Further, because the process associated with many outcomes is at least as important as the resultant product, process assessments will have increased importance. For example, effective teamwork has more to do with the students' interactions and teaming capabilities (given the environment and purpose) and less to do with the resultant product. This transition towards the development of more in-depth assessment techniques has already begun as evidenced by a recently developed NSF research agenda for engineering education which includes assessment among its five research areas [12].

Particularly interesting here is behavioral observation, an area of assessment that can capture aspects of those processes students use in mastering various engineering outcomes. A rich, in-depth assessment method such as behavioral observation [13, 14] is desirable because it enables us to investigate engineering learning in-action and in real time. Further, it holds the potential to evaluate students at the higher level learning domains. However, even though 100 percent behavioral observations are effective at capturing process oriented outcomes (i.e., teamwork, resolving ethical dilemmas, etc.), they are expensive (in terms of effort and resources) to administer. One potential remedy is the use of statistically based methods, such as work sampling, to assess process oriented outcomes at substantially less cost than full behavioral assessment.