Effectiveness of Challenge-Based Instruction in Biomechanics

Journal of Engineering Education, Oct 2006 by Roselli, Robert J, Brophy, Sean P

B. Test for CBI Instructor Bias

Average student evaluations for both instructors, teaching BME 101 in the traditional mode, are shown in the first three columns of Figure 3. Comparisons of student perceptions of amount learned, intellectual challenge, and overall course evaluations are shown in the last three columns. Clearly, there is no advantage provided by the senior instructor over the junior faculty member in any of the categories. In fact, student ratings are generally better for the sections taught by the junior faculty member. Therefore, we infer from these student evaluations that the instructors are both competent instructors with equivalent classroom experience from the students' perspective when using traditional instruction.

C. Test for Semester Bias

Student performance on three free body diagrams (FBD) completed during the first week of class showed no difference in means (p = 0.74) between traditional (58.1) and CBI (57.1) classes. When performance on each individual FBD was compared, the traditional class did slightly better on the first and the CBI class did slightly better on the third FBD problem, but these differences were not significant (Figure 4). These results show no systematic differences and suggest that students entering BME101 in the Spring semester are no better prepared in the fundamentals of mechanics than students entering in the Fall.

D. Student Performance

The question remains: "Does participation in a challenge-based course improve students' abilities to solve problems?" Or, is the CBI approach just an alternate version of the traditional course that may improve student attitudes, but not their performance? We compared performance by students in CBI and control courses on 50 final exam questions asked over a three-year period. The CBI group outperformed the control group on 26 percent of the questions, while the control group outperformed the CBI group on only 8 percent of the questions (p

Several of the questions were relatively easy, and it is unlikely that a comparison between the two groups on these questions would show any differences. We subdivided the questions into two categories wherein the average percentage of correct rubrics for a particular question by all students was either greater than or less than the average for all students on all questions (i.e., average = 60.7). Twenty-seven of the questions fell into the "less difficult" category and 23 questions composed the "more difficult" category. A comparison of these results can be found in Figure 5. There was no significant difference in means between the CBI (78.4) and control (79.0) groups for the easier questions (p = 0.33). However, as shown in Figure 5, the CBI group (average = 42.6) scored significandy higher (p = 0.02, paired Student t-test) than the control group (average = 36.6) on the more difficult questions. The CBI group performed better than the control group on 35 percent of the more difficult questions, while the control group managed to outperform the CBI group on only 4 percent of those questions (Table 2). The effect sizes for the two categories of questions are shown in Figure 6. Those questions with significant differences are designated with an asterisk Questions in which students in CBI classes did much better than students in traditional classes (i.e., ES >1.0) dealt with more difficult concepts. These questions included free body diagrams that involve couples, bending questions requiring the computation of the centroid of a trapezoidal member, determination of the center of gravity of a composite body, and the prediction of the mode of failure in bending.

Another way to examine the effectiveness of the HPL approach is to compare performance between classes on individual rubrics, rather than performance on the problem as a whole. As an example, in the last year of the study we asked students in the CBI course to construct free body diagrams for three different systems at the beginning of the semester. We then had students in both classes generate the same free body diagrams at the end of the semester. As expected, students performed better at the end of the semester than they did on the first day of class. This is illustrated for one problem in Figure 7a (p

A second example of the use of a detailed scoring rubric to discover differences in performance is illustrated by the problem in Figure 8. Although there is no difference between the two classes on total score (sum of all rubrics), it is clear that the most difficult concepts for students in both classes are inclusion of a couple at point B and inclusion of the couple in the sum of moments equilibrium equation. Only 20-30 percent of the students in the traditional course got these portions of the problem correct, while 40-50 percent of those in the challenge-based class got them correct.