Improving Learning in First-Year Engineering Courses through Interdisciplinary Collaborative Assessment

Journal of Engineering Education, Jan 2008 by Qualters, Donna M, Sheahan, Thomas C, Mason, Emanuel J, Navick, David S, Dixon, Matthew

Given these requirements, an assessment mechanism was needed that would provide an objective measure of the quality of preparation of first-year engineering students to continue into the second year with its intensive engineering courses that require the skills of application, analysis, and, evaluation of the learned concepts. While student grades were helpful, they did not indicate with any precision the areas that students have mastered and retained after the course was completed.

We had to distinguish between surface and deep learning which are critical for academic advancement. These can be characterized by the students' approach to the task of learning course material. Surface learning can be described as using low-level cognitive skills and minimum effort to complete course requirements. On the other hand, deep learning involves understanding, engaging in higher-level cognitive skills, making connections within and across disciplines, and thinking conceptually about a topic (Cox and Clark 1998).

To make the assessment of learning in the first-year engineering curriculum truly meaningful, a number of factors needed to be in place. First, the form of the assessment chosen needed to match the objectives of the first year courses. This required that the assessment mechanism be constructed by the appropriate discipline's faculty. In addition, the selected assessment tools had to assess deeper levels of learning. The measurement instrument had to be carefully crafted to elicit each student's ability to apply and evaluate information in different contexts (Banta, 1993).

As noted in Leskes and Wright (2005) and implied in the ABET engineering criteria, a good assessment process requires that information be fed back to the department to promote understanding of what the student cohort learns over time, and which areas of the course may not have been learned as well as others. Additionally, since individual results on each item are reported to the students who took the exam, the results help them self-assess the quality of their knowledge and their ability to apply course concepts.

We will describe the process of developing a diagnostic mastery exam administered to second-year engineering students to assess competency gained by students in the first year. We will describe the analysis of the exam results using Item Response Theory (IRT), and the use of those analyses as a basis for providing feedback to the faculty members teaching those subjects. Furthermore, we will describe how the feedback process led to constructive dialog on curriculum changes, teaching methods, and course objectives and outcomes, and promoted a collaborative environment among the various faculties responsible for teaching FYES.

II. INITIAL EXAM DEVELOPMENT

A committee was appointed to serve as an exam development team, consisting of the principal investigator and the co-principal investigator from NUs College of Engineering (COE), director of the university's Center for Effective University Teaching (CEUT), instructors from the four departments (Chemistry, Computer Science, Math, and Physics), a faculty member from the Counseling and Applied Educational Psychology (CAEP) Department whose background was in psychometric assessment, and a consultant to assist with data analysis. The project team went through a series of educational workshops and working sessions to prepare for development of a valid assessment exam that would most accurately measure student learning across four discipline areas taught during the first year. For each course in which questions were to be developed, the tasks included developing course objectives and learning outcomes, constructing tables of specification (showing learning levels to be assessed within Bloom's taxonomy, and their relative importance), and writing the actual multiple choice questions.