Development of a Work Sampling Methodology for Behavioral Observations: Application to Teamwork

Journal of Engineering Education, Oct 2007 by Besterfield-Sacre, Mary, Shuman, Larry J, Wolfe, Harvey, Clark, Renee M, Yildirim, Pinar

The results obtained from testing work sampling showed that it was a viable methodology for all four environments. Table 6 summarizes the results obtained from each of the four learning environments. Target values and lower and upper 95 percent confidence limits for each subject and teamwork category were obtained; and two proportions for each subject/category combination during work sampling were acquired. For the T-Shirt Design environment, of the 48 work-sampled proportions, only eight proportions fell outside the 95 percent confidence intervals established around the target values.

The results obtained from the Delta Design environment were even more promising than the results from the T-Shirt Design environment. Of the 64 work-sampled proportions, 60 (93.8 percent) fell inside the 95 percent confidence intervals established around the target values. The work sampling results for the regression environment were similar in accuracy to those of the Delta Design environment. Of the 48 work-sampled proportions, 45 (93.8 percent) fell inside the 95 percent confidence intervals established around the target values.

For our final learning environment, the capstone project, it was infeasible to 100 percent behaviorally observe a semester's worth of work Instead, we randomly selected several 30-minute segments of the term and tested them in a similar manner to the other three. Work sampling results were complementary to the other environments tested with only three (6.3 percent) of the 48 falling outside the confidence interval. That is, for the four environments, and 208 intervals, only 18 (9.1 percent) fell outside the confidence limits.

VI. DISCUSSION AND ADDITIONAL WORK

This paper has described our methodological approach to develop and validate work sampling of project-based teamwork. Our methodology, termed Cognitive, Affective, Behavioral Sampling (CABS), has been shown to successfully provide an effective mechanism for capturing cognitive and behavioral information currently obtained through expensive 100 percent behavioral observation. Based on the validation results from various project-based learning environments, this work sampling methodology could be used by educators as the foundation for a cost-effective assessment system for teamwork, while holding the potential to be applicable for other process oriented outcomes. To do the former, a next step would be to also assess the quality level at which each teamwork attribute is being performed, most likely by using rubrics, a methodology that has been used successfully by others. In addition, one needs to relate the quality of the actual output (project result) to both the quality of each attribute and the proportion of time devoted to that attribute. Statistical models could be built to do this further validation, which is a research direction we hope to pursue in the future as discussed below. However, to address the latter, the methodology needs to be tested for other outcomes to see if it is equally as effective as for teamwork, as also discussed below.