Assessing pharmacy student knowledge on multiple-choice examinations using partial-credit scoring of combined-response multiple-choice items

American Journal of Pharmaceutical Education, Spring 2000 by Wongwiwatthananukit, Supakit, Popovich, Nicholas G, Bennett, Deborah E

This study developed, implemented, and evaluated the use of partial-credit scoring for the combinedresponse multiple-choice (CRMC) examination format utilized in an elective nonprescription drugs course. Behavioral objectives and CRMC questions were developed and classified according to the appropriate cognitive domain of Bloom's Taxonomy of Educational Objectives, External and internal content panels validated these objectives and reviewed developed CRMC questions prior to their use on three course examinations. Item response data for the partial credit scoring and the traditional dichotomous scoring method were then compared in terms of item difficulty, item discrimination, reliability, efficiency, cognitive domain being measured, and criterion-referenced grading assignments. The results demonstrated that the partial credit scoring method increased item difficulty and resulted in student outcomes which were more reflective of student knowledge compared to dichotomous scoring. Further, the partial credit scoring method increased the reliability and efficiency when compared to the dichotomous scoring method. The item difficulty difference attained by using the two methods of scoring was statistically different (P

INTRODUCTION

In pharmaceutical and other health professional education, the most commonly used assessment instrument is the test consisting of multiple-choice (MC) items, particularly the singleanswer (SA), four or five foil multiple-choice item and the combined-response multiple-choice (CRMC) item which is one type of complex multiple choice (Type K) item. CRMC items are combinations of three primary responses, and the combinations of the primary responses are referred to as secondary choices. The secondary choices which are specific for the choice of the three primary responses serve as options. A typical CRMC format is illustrated by the following example:

Which of the following laxative products is(are) appropriate to recommend to a man in otherwise good health who suf fers from painful hemorrhoids?

1) Dulcolax Tablets

2) Fiberall Powder

3) Metamucil Packets

Response Format: A. 1 B. 3 C. 1,2 D. 2,3 E. 1,2,3

Answer: D

This format, compared to the traditional single-answer, four or five option multiple-choice item is used to test student achievement within the pharmacy curriculum courses and is also utilized in the North America Pharmacist Licensure Examination

(NAPLEX) to evaluate the minimal competency of the pharmacy graduate to enter the profession of pharmacy. An advantage of the CRMC format is that it accommodates those occasions when more than one choice/response is correct. Indeed, within pharmacy practice, there are many occasions when more than one response is correct and/or appropriate. Oftentimes, too, in practice, there are also degrees of correctness (i.e., most correct, somewhat correct) when responding to certain questions/inquiries. Some researchers recommend against using the complex multiple choice (Type K) format for several reasons(1-3). For example, complex multiple choice items are more difficult than single best answer multiple choice. This format requires more time to read and could affect the number of items used in a test (i.e., sampling of content). This format may produce items with lower discrimination which could affect test score reliability. This format is dif ficult to construct and edit. Although there are several disadvantages of this format, it is important to note that the complex multiple choice format that these researchers criticized is not the one used in this study.

Historically, the scoring of multiple-choice examinations has been problematical. Dichotomous scoring consists of assigning a certain point value for a correct answer and zero points for an incorrect answer. It causes inefficiencies in the interpretation of test scores because the response to a MC item may come from different information levels (i.e., complete information, partial information, misinformation, and total ignorance)(4-6). Complete information is described as the knowledge that allows that student to select the most correct response and most incorrect answer from among presented options in a MC item(5). Partial information is described as the information that students have which allows them to recognize some of the incorrect response options as clearly wrong while being unable to recognize all of the correct options(5-7). Partial information, in itself, is not incorrect information. However, it is incomplete knowledge, differing from misinformation in being only partially correct, rather than completely incorrect. If a student believes the correct response is wrong without guessing then the student has misinformation. Incorrect options may also be eliminated due to misinformation(5).

In test scoring of the students with partial information or misinformation, the ability to discriminate degrees of correctness or incorrectness has usually been ignored. Students are given no credit for rejecting the worst responses or choosing somewhat correct responses. This can be observed from the example CRMC question above. The correct answer is "D." Based upon conventional dichotomous scoring, students who select "D" would get full credit and all other responses would be valued at zero credit. But, students who selected "B" are also partially correct, even though they did not select the most correct response. This demonstrates how dichotomous scoring cannot reward students for less than full knowledge or mastery of the content area.