A Paradigm for Assessing Conceptual and Procedural Knowledge in Engineering Students

Journal of Engineering Education, Oct 2007 by Taraban, Roman, DeFinis, Alli, Brown, Ashlee G, Anderson, Edward E, Sharma, M P

ABSTRACT

Conceptual and procedural knowledge are two mutually-supportive factors associated with the development of engineering skill. The present study extends previous work on undergraduate learning in engineering to provide further validation for an assessment paradigm capable of quantifying engineering students' conceptual and problem-solving knowledge. Eight students who were enrolled in an introductory thermodynamics course and four who were enrolled in the course sequel provided verbal protocol data as they used instructional software. They were compared to existing data from a cohort of eleven science and engineering majors who had not taken thermodynamics. The results replicated earlier findings showing more cognitive activity on computer screens requiring overt user interaction compared to text-based screens. The data also indicated that higher- versus lower-performing students, based on course grades, engaged in more higher-order cognitive processing. There was no evidence that students gained deeper cognitive processing as they advanced through the engineering curriculum.

Keywords: cognitive processing, instructional software, skill development, assessment

I. INTRODUCTION

A pressing goal of engineering education today is to find ways to draw more students at all levels into the culture and practices of engineers. In order to do this, we need innovative assessments capable of telling us in more detail about how students approach and think about engineering concepts and problems. This position is consistent with the conclusions of other researchers:

Conventional metrics such as standardized examinations or accumulation of credit hours are no longer adequate to assess fully the complex outcomes of engineering education. Instead, we need measures that examine the qualitative changes in students' thinking processes [1, p. 39].

The goal of the this research was to extend our work on the development of an assessment paradigm capable of quantifying engineering students' conceptual and problem-solving knowledge [2]. The development of this assessment tool has drawn on several contemporary theories and approaches to learning.

A principle of constructivism in student learning [3] has been adopted by researchers in engineering education. A constructivist approach considers the preconceptions [4] that students bring to the learning situation, because students build on what they already know. Through active, hands-on learning [3, 4], students extend and refine even more adaptive cognitive representations and associated skills in their domain of training. Effective problem solving [5] is closely associated with concept learning, making inferences, and categorization [6], which represent distinct components of engineering skill.

When students understand a concept or problem, they do so along a continuum that can be characterized as extending from shallow to deep knowledge. A distinction between shallow and deep knowledge has been well articulated in the research literature on text representation and comprehension [7,8].

Shallow knowledge consists of explicitly mentioned ideas in a text that refers to: lists of concepts, a handful of simple facts or properties of each concept, simple definitions of key terms, and major steps in a procedure (not the detailed steps). Deep knowledge consists of coherent explanations of the material that fortify the learner for generating inferences, solving problems, making decisions, integrating ideas, synthesizing new ideas, decomposing ideas into subparts, forecasting future occurrences in a system, and applying knowledge to practical situations. Deep knowledge is presumably needed to articulate and manipulate symbols, formal expressions, and quantities, although some individuals can master these skills after extensive practice without deep mastery [7, p. 6].

The most prominent outcomes of deep knowledge are longer-term retention of information due to more elaborated cognitive representations of the knowledge, and significant advantages in transferring the knowledge to novel situations because the knowledge is not tied to specific rote situations and procedures. Classic studies on the development of expertise in physics showed that novice undergraduates are easily misled by surface problem features. For instance, when given a sorting task, they readily sorted together problems involving inclined planes, or problems involving pulleys, with little regard for the underlying principles involved in the problem (e.g. , conservation of energy) that would allow sorting on a deeper, more meaningful level [9]. When given a story problem, novices are also known to "work backward" from the unknown variable value, patching together equations that come to mind involving that variable [10; see also, 11], whereas experts show forwardthinking reasoning, categorizing the problem and identifying the relevant physical principles as they read through the problem. Experts pay attention to what information is already given in a problem, and they anticipate what they might have to calculate and possible ways of carrying out those calculations, in advance of actually pursuing a specific solution.