Development of a Work Sampling Methodology for Behavioral Observations: Application to Teamwork

Journal of Engineering Education, Oct 2007 by Besterfield-Sacre, Mary, Shuman, Larry J, Wolfe, Harvey, Clark, Renee M, Yildirim, Pinar

Relative to cost, the time savings associated with the work sampling procedure was substantial. It took observers an average of 1.5 hours to conduct 100 percent behavioral observation for a 30-minute taped interval. In contrast, for the same 30-minute interval, it took less than 30 minutes to conduct work sampling to achieve the same information.

V. RESULTS

To further validate work sampling for the teamwork outcome, four different learning environments were tested. These corresponded to undergraduate engineering courses requiring projects of different duration and intensity. These environments were representative of project-based learning environments involving teamwork. Environment 1 involved a single lab project in which the team of students had to design a T-shirt. Environment 2 involved a short project-Delta Design (developed and copyrighted by Professor Louis Bucciarelli at MIT [42]). This project has been used by various engineering programs to introduce design and teaming principles by having students design a shelter for a hypothetical, two-dimensional planet. In environment 3, a second short project involved an empirical modeling assignment as part of a probability and statistics course. In the fourth environment, teams of students worked on a multi-disciplinary capstone design project incorporating multiple courses and skills. The capstone project was part of a product realization course in which the students took a product from concept to prototype and business plan. These environments are summarized in Table 5.

Two student teams served as subjects for each of the four environments. The subjects were undergraduate engineering students who volunteered to participate and were compensated for their involvement. The use of two teams for each environment accounted for variations in group capability and performance and the potential effects on the sampling results. Each team completed its project work in a dedicated work space with video equipment that captured all action in the space.

To summarize, the goal was to obtain statistically similar work sampling results to those for 100 percent observation (the target values). The tapes for each learning environment were used to 100 percent observe each team member for approximately 90 minutes to establish target values for each teamwork category. A different group of observers then work-sampled the individual team members using a floating-lengdi interval. Observation samples were taken approximately every two minutes based on the calculation from the work sampling experiment ((1). Inter-rater reliability was established before setting the target values. ANOVAs within each environment provided evidence that the observers were reliably interpreting events, thereby allowing us to pool results; i.e., take averages of the proportions for each subject and teamwork category combination.

We conducted three-factor ANOVAs with the factors being observer, time period, and teamwork categories, each of which was thought to contribute to the overall variance. The ANOVAs showed no significant differences between any pair of observers when observing each subject. The time period factor was usually insignificant as well. However, the teamwork categories were significant, as we would not expect to observe each category equally oneeighth of the time.