Improving Learning in First-Year Engineering Courses through Interdisciplinary Collaborative Assessment

Journal of Engineering Education, Jan 2008 by Qualters, Donna M, Sheahan, Thomas C, Mason, Emanuel J, Navick, David S, Dixon, Matthew

In retrospect, using quantitative results as feedback proved to be a useful format for the target group of instructors since all were from analytic or technical backgrounds (math, science, or engineering). More importandy, for implementing the feedback process, this quantitative format proved to be a means for focusing on the results and their interpretation related to learning, rather than on teaching. This is important since there had traditionally been a somewhat adversarial relationship between the COE administration and departments responsible for first-year courses in engineering; this relationship often was based on summative judgments of teaching quality. This concept will be explored further in the section on feedback Therefore, the workshop focused on design of multiple choice questions. The working group agreed that there should be an equal number of alternatives for each question, concluding that four is the ideal number (the correct answer and three distractors). A larger number would add to the development effort and could potentially reduce the clarity of the results; a smaller number might not provide sufficient discrimination. Tables of specifications (TOS) for the tests in each of the four academic areas were then prepared by the group. These were reviewed and revised in subsequent meetings. Table 1 provides a sample of one of the tables of specification, in this case from Physics 1. One or more "X" marks indicates the degree of relative emphasis for the particular topic. This exercise further promoted continuing dialogue around curriculum as each discipline instructor was able to review the TOS from the other disciplines. In some cases, faculty were unaware that certain topics were not addressed in courses or that their course was expected to emphasize areas engineers deemed crucial for successful continuation of students in year two.

Once the proposed exam items were developed, the working group went over each item and made suggestions to improve clarity and other aspects of item design. Thus, the exam was developed by the interdisciplinary team promoting an atmosphere of group ownership of the exam and its validity. This proved to be an important feature for using the exam results for feedback to the instructors.

III. ADMINISTRATION OF THE EXAM IN FALL 2004 AND FALL 2005

In September of 2004, the four-part, 45-item exam was administered in an online format in computer classrooms to 191 full-time, matriculated second-year students in civil, mechanical, and chemical engineering. The idea behind giving the exam in the second year rather than at the end of the first year, is that it would assess more accurately what students have retained over time from their first year courses in chemistry, computer programming and applications, math, and physics. In other words, the exam would determine how much of the essential knowledge and skill set students carry forward to apply in follow-on engineering courses. In September 2005, the revised exam, with 46 items, was again administered in the same manner to the next class of second-year engineering students in the same three majors for a total of 221 students.