Evaluation of instructional design of computer-based teaching modules for a manufacturing processes laboratory

Journal of Engineering Education, Jul 2000 by Hailey, Christine E, Hailey, David E

During the fall term, 1996, 38 students were randomly divided into two groups, each half was assigned either Module A or B. Students worked their ways through the modules filling out a worksheet as they progressed. This helped insure that all students visited all available information. Once they completed the module, students took a quiz to assess their understanding of the material. We also observed the students working with the modules and noted that students using the author-driven module typically took approximately half the time of students using the reader-driven module.

Again, because of group size we used a t distribution. We determined that there was a significant difference in the average scores of the two groups at 0.01 level of significance. The students working with Module B did worse, on the average, on the quiz. To ensure we had randomly selected groups, we once again looked at their final grade in the course. There was no difference in the average, final grade of the two groups to a 0.01 level of significance.

We found these results surprising and decided to take a closer look at the differences in learning between the two groups. Breaking these two large groups into smaller groups, based on grade points, was not part of our original experimental design and so the following results should be viewed as preliminary. If we break the students out by the final grades they received in the class, we discover that "A" students in both groups show no statistical difference in performance on the quiz. "B" students, on the other hand, missed an average of 1.2 questions on Module A as opposed to 4.2 from Module B-this represents a statistically significant difference on average test scores at a 0.05 level of significance. "C" students studying from Module A missed an average of 2.6, while "C" students studying from Module B missed an average of 4.5 questions-this represents a statistically significant difference on average test scores at an 0.2 level of significance. Given this high level of significance, the comparison of the grades for the "C" students indicates a trend at best. However, in trying to understand learning differences this is valuable information.

These results imply that there is a difference between information acquisition in reader-driven compared with author-driven instructional design. It appears "A" students work equally well in either environment. Although Wallace and Mutooni did not observe whether their students used the web-based module in an author- or reader-driven fashion, we suspect it did not matter. The students enrolled in their program would be "A" students at USU. However, we were interested to note that "B" and "C" students (at USU, a "B" student is an average student and a "C" student is a below-average student) appear to learn the material better in an author-driven environment. The above study has important implications for developing CBT tools. If all students must absolutely understand material, e.g., laboratory safety, the CBT should be developed as an author-driven tool. On the other hand, these results suggest that "average" and "below average" engineering students may be very task-oriented, linear thinkers. If the educator's goal is to develop nonlinear, open-ended problem solving skills, a reader-driven tool may be more effective. Or, possibly, a hybrid scheme, where the student is initially exposed to an author-driven environment and then the control is slowly transitioned to a reader-driven environment. At a minimum, these results indicate more research on the nature of engineering students' responses to various CBT instructional designs is important.