Improving Learning in First-Year Engineering Courses through Interdisciplinary Collaborative Assessment

Journal of Engineering Education, Jan 2008 by Qualters, Donna M, Sheahan, Thomas C, Mason, Emanuel J, Navick, David S, Dixon, Matthew

Figure 3 shows the results of modification of Item 14 from the math (calculus) section, which in 2004 provided fairly poor discrimination, and even high ability students (θ=3) had less than a 75 percent chance of answering this correctly. After the question was modified with a suggestion to draw the problem in 2005, the item was a better discriminator (higher slope at P = 50), and high ability students had a 96 percent probability of answering it correctly.

Finally, Figure 4 shows perhaps the most dramatic change due to a question revision for Item 39 from physics. This involved collisions of two rigid balls, and the problem was revised from 2004 to 2005, resulting in a change in the correct answer from "a" in 2004 to "b" in 2005. This revision led to a dramatic change in the ICC for the item, resulting in a highly discriminating item with reasonably high difficulty level.

V. DISCUSSION

A comprehensive mastery exam analyzed using Item Response Theory to assess student learning provided a probabilistically defensible method to measure how much engineering students have learned and retained from their critical, but often uneven, classroom experiences during the first year. The mastery exam items were formulated using Bloom's taxonomy, thus assuring items covered measurement of learning to the application level deemed appropriate for first year courses. The online delivery of the exam allowed students to receive immediate feedback on how they performed on each question. This helps students to understand knowledge areas in which they need to focus attention in future course work. This method also allowed us to provide feed back to the host departments (Chemistry, Computer Science, Math and Physics) about how well students were grasping key concepts that are essential to their follow-on engineering studies.

In terms of the relationship between the COE and the discipline departments that teach these first-year courses, the use of IRT helped to create a more objective, data-driven mechanism to serve as a foundation for reflection and improvement in the host departments. This reflection included discussions about how the various instructors are teaching a topic, particularly when taught across multiple sections. How effectively the topics were being taught is related to what is being taught and the mastery exam results often led to greater uniformity across sections, and in some cases, curriculum changes within one of the courses evaluated in the exam. These various aspects of feedback are the essence of the assessment process, which is critical to ABET engineering accreditation and is now gaining further acceptance in the general education arena (e.g., Leskes and Wright, 2005). Because the host departments took ownership of this feedback process, the relationship between the COE and these departments became less adversarial, and an atmosphere of quality improvement began to replace this traditional point of tension.

With further analysis, the data from annual exam administrations would also allow longitudinal comparisons to be made from one engineering class to the next, which can be correlated to other more common metrics being used to assess student quality at an institution.