Improving Learning in First-Year Engineering Courses through Interdisciplinary Collaborative Assessment

Journal of Engineering Education, Jan 2008 by Qualters, Donna M, Sheahan, Thomas C, Mason, Emanuel J, Navick, David S, Dixon, Matthew

ABSTRACT

This paper describes a feedback process that assessed first-year engineering student learning using a mastery exam. The results were used to improve learning and teaching in first-year courses. To design the initial exam, basic knowledge and concepts were identified by instructors from each of the host departments (Chemistry, Math, Physics and Computer Science). In 2004, the 45-item exam was administered to 191 second-year engineering students, and in September 2005, the revised exam was administered to the next class of second-year engineering students. The exam was analyzed using Item Response Theory (IRT) to determine student abilities in each subject area tested. Between exam administrations, workshops were conducted with the four department instructor groups to present exam results and discuss teaching issues. The exam provided a learning assessment mechanism that can be used to engage faculty in science, madiematics, and engineering in productive linkages for continual improvement to curriculum.

Keywords: assessment, faculty development, first-year students

I. Introduction

While it can be argued that student learning has always been the goal of higher education, emphasis on understanding and measuring student learning, rather than teaching, is a much newer phenomenon. The shift to the concept of student-centered learning (Barr and Tagg, 1995; Huba and Freed, 2000; Jafari, 1999) and the concurrent agenda focusing on outcome assessment have increased the pressure to quantify student learning beyond the assignment of course grades. The recent Spellings Commission Report on the state of higher education (U.S. Department of Education, 2006) reinforces the need for disciplines to develop measures to document the learning of their students. Research suggests that universities may not be as successful at educating students as once thought, and grades used to support these claims of success may reflect inflated metrics (Bok, 2006; Gradeinflation.com, 2006; Rojstaczer, 2003).

The engineering fields have responded with the reformed ABET engineering accreditation criteria that emphasize the need to systematically monitor, validate, and subsequendy improve the learning that students receive in their course of study. As engineering faculty know all too well, gone are the days when accreditation consisted of "bean counting" and using easily identifiable numbers such as credits offered in particular topics. The old approach has been replaced with the requirement to demonstrate program quality that students have knowledge of their discipline at the higher levels of learning beyond the course grade. This has become the measure of programmatic quality.

Although the mandate for documentation of learning quality is clear, the methods for doing this are not as obvious. How does one demonstrate the knowledge and skills students actually retain, integrate, and use in doing engineering? Further, given the ABET emphasis on feedback loops for improvement, how can a process be put in place to achieve this feedback and improvement? This problem is particularly vexing when it comes to assessment of first-year engineering student (FYES) learning. In many institutions, the first-year course work is provided outside of the engineering departments. Students may be taking these courses from a wide variety of instructors covering topics prerequisite to the study of engineering such as chemistry, math, and physics, among others. The learning objectives and skills taught in these courses are more often geared to the specific discipline rather than to the knowledge and skills students must have to continue their engineering education. Research on learning in science, technology, engineering, mathematics (STEM) disciplines suggests that the ability to problemsolve comes from the creation of increasingly complex connections among diverse learning sources (Bransford, Brown, and Cocking, 1999; Donald, 2002). But how do we know that students have grasped material from first-year courses as they enter successive phases of their engineering education?

Previous work at Northeastern University (NU) on improving the relationship between the Arts and Sciences faculty and those from Engineering produced a retention exam that was administered to FYES at the end of their first year. We quickly realized the inadequacy of this measure. First, we were not really measuring knowledge retention, but rather what students had studied to prepare for final exams. We needed to understand what knowledge was retained over the summer months and would be available in their intellectual tool box when they returned in the fall for their specific engineering courses in the major. Second, we had not used the results of the retention exam in any significant way to change curriculum. Students were given their scores but no attempt at dissemination or correlation with learning outcomes was attempted. To change the curriculum to synchronize the needs of the upper level engineering curricula with the preparatory courses in the first year, a feedback loop was needed. Lasdy, while this previous initiative had opened dialogue among faculty teaching FYES and the engineering faculty, we wanted to further develop the positive relationship. We were interested in transcending discipline differences to focus on what engineering students had learned in the first year that would allow them to be successful in year two. This led to the design of a study to identify the learning that was retained over the summer and to conceptualize a faculty development method to present this information to departments that would engage them in serious discussion around student learning and the need for improved teaching.