Comparison of traditional testing methods and standardized patient examinations for therapeutics

American Journal of Pharmaceutical Education, Fall 2001 by Gardner, Stephanie F, Stowe, Cindy D, Hopkins, Denise D

Standardized patient (SP) examinations require that a student demonstrate problem solving, comprehension, and communication skills. The primary objective of this study was to determine if there is a correlation between performance on SP examinations and traditional testing methods in a therapeutics course. A secondary objective was to compare three standard-setting procedures for an SP examination. A final examination, consisting of a 75-item multiple choice examination and a three-station SP final examination, was administered to 73 pharmacy students. Three standard setting procedures (Angoff, borderline, and holistic) were applied to the SP examination data. A moderate positive correlation was seen between performance on the multiple-choice and SP final examinations. There was a weak correlation between performance on interim examinations in the three content domains and performance on the individual SP cases related to that domain. The Angoff and borderline procedures gave similar results, though the borderline method provides a less labor-intensive approach. An SP final examination provided additional information regarding students' problem-solving, clinical reasoning, and communication skills at the culmination of therapeutics.

INTRODUCTION

Students in most medical schools now undergo standardized patient (SP) based assessments(l). The United States Medical Licensing Examination has done extensive research to determine the feasibility of incorporating SP examinations into their licensing procedures for physicians, in a manner similar to the licensing procedure for physicians in Canada. SP examinations are now a compulsory part of the licensing examination for Canadian pharmacists, through the Pharmacy Examining Board of Canada (PEBC)(2). These examinations are also used as a part of continuing competency testing for practicing pharmacists in the province of Ontario(3). Although this form of assessment is not routinely used for licensing or continuing competency assessment for pharmacists in the United States, the emphasis on competency-based curricula and evaluation by the American Council on Pharmaceutical Education (ACPE) stresses the need for continued research in this area. ACPE recommends clinical evaluation measure "cognitive learning, mastery of essential practice skills, abilities to communicate effectively and to use data in the critical thinking and problem solving processes"(4). The objective structured clinical examination (OSCE) is a method that utilizes SPs to measure clinical competence by focusing on outcomes through observable behaviors.

Although preparing students to become health care providers who are capable of providing pharmaceutical care in a variety of practice settings is the primary mission of pharmacy education, the consistent measurement of a student's ability to provide this type of care remains elusive. The problem solving skills and communication skills that must accompany a sound knowledge base are difficult to assess. Recently, the American Association of Colleges of Pharmacy (AACP) proposed that the Pharmacy College Admissions Test (PCAT) include an essay component to evaluate critical thinking skills(5). The AACP PCAT Advisory committee recommended an essay component, rather than a multiple-choice format, secondary to its ability to not only assess critical-thinking, but also writing skills. The perceived need to assess critical- thinking and communication, even prior to admission, is evidence of the shift toward ability-based outcomes measurement in pharmaceutical education.

The objectives of this study were to compare the scores on individual short-answer interim examinations and corresponding cases on the standardized participant portion of the final examination, as well as to compare the scores from two parts of a final examination: a 75-item multiple-choice portion and a three-case standardized patient based portion. A secondary objective of the study was to compare three standard-setting procedures for a standardized patient examination for pharmacy students.

METHODS

Student Selection

All students enrolled in Therapeutics I participated in this evaluation. Therapeutics I is a five-hour required course in the spring semester of the second professional year of the curriculum. The focus of Therapeutics I is the use of drug therapy in treating major disease states, including a review of pathophysiology of the disease, therapeutic goals, plans of treatment, dosage regimens, therapeutic alternatives, and therapeutic endpoints. The course consists of four 50-minute lecture hours weekly, in addition to a weekly recitation laboratory in which the class is divided for small group discussion and case presentation. A total of 74 students were enrolled in the spring semester of 2001.

MC Examination

A 75-item multiple-choice computerized examination was administered in a two-hour block to students in Therapeutics 1. This examination was scheduled three days after the SP examination during finals week.