Improving Learning in First-Year Engineering Courses through Interdisciplinary Collaborative Assessment

Journal of Engineering Education, Jan 2008 by Qualters, Donna M, Sheahan, Thomas C, Mason, Emanuel J, Navick, David S, Dixon, Matthew

A. Use of Item Response Theory for Exam Results Analysis.

The exam results were analyzed using a model based on Item Response Theory (IRT). Unlike classical test analysis, which simply adds the raw score from each test item to produce a student's score, IRT is answering two questions simultaneously (Embretson, 1999):

1) Given that a student has some true level of knowledge in a certain subject, what are the qualities or characteristics of exam items that would give a true measure of this level of knowledge?

2) Given the set of items in an exam, what is most likely to explain the student's performance on that exam?

Thus, the IRT analysis is one that is highly iterative, with the solution coming from the maximum likelihood that both of these questions have been answered with the highest level of probability possible to obtain from this set of test data. We chose this methodology because it would help to achieve our original goal of assessing student learning with some level of objectivity, and would also indicate the quality of our exam questions.

While the mathematical background for IRT is beyond the scope of this paper, the analysis is based on a logistic model (Hambleton, Swaminathan, and Rogers, 1991) that has as its inputs the student responses to each item on the exam. The output is a probability (P^sub i^) that a randomly chosen student with an ability level θ in a topic area will answer a particular item i correctly. When P^sub i^ is plotted versus 6 for a particular item, the result is item characteristic curve (ICC). Figure 1 shows four conceptual ICCs to illustrate graphically the significance of the item characteristics.

For the analysis of the mastery exam, a three-parameter model was used to represent the ICC. For an item, i, the parameter a, quantifies the item's effectiveness at discriminating among abilities, and is the slope of the ICC where it crosses the P(θ) = 50 percent level. For example, a relatively high slope at P(θ) = 50 percent (corresponding to a relatively high value of a^sub i^) indicates that P^sub i^ changes significantly at some ability level, which would make the item valuable for differentiating abilities. Referring to Figure 1, the two solid line graphs (marked 1 and 2) have relatively high slopes at P(θ) = 50 percent indicating that these two questions are better discriminators compared to the dot-dash line (marked 3), indicating that these two questions are good discriminators. On the other hand, the parameter b^sub i^ is the ability level where the ICC crosses the P(θ) = 50 percent level, and is used to represent the item's difficulty. In Figure 1, the difficulty levels of the two "good discriminator" items (marked 1 and 2) are very different, and the "poor discriminator" item (marked 3) has a difficulty in between those two. The value of c^sub i^, known as the pseudo-chance-level parameter (or "guess factor"), is a lower bound asymptote for the ICC. From a practical point of view, this indicates the probability that an examinee of lower ability can answer the item correcdy. However, in an exam designed to assess the degree of mastery of a subject, it is desirable to have some items that may produce relatively high c^sub i^ values. This would suggest that minimally acceptable levels of mastery were achieved by a large number of examinees; the ICC for such an item is shown in Figure 1 by the dashed line (marked 4). Thus, for each question or item on the assessment exam, an ICC is produced that can provide a snap shot view of the item difficulty, its quality in terms of its discrimination among ability levels, and its use as a mastery-type question.