Assessment and outcomes in medical education

Military Medicine, Sep 2003 by Pangaro, Louis

Thank you very much for the chance to talk about assessment and evaluation. I will first try to use precise terminology and definitions to increase our understanding of issues in evaluation and, second, focus upon an actual product from the conference. My emphasis will be on how to perform assessments.

Let me discuss the goals for evaluation. Fairness is I think the heart of evaluation. The stakes are high for patients and they are high for students and residents. We have to be fair, but the first group to whom we have to be fair comprises those in uniform, our patients, and the community. So, there has to be a clearer demonstration of trainees' competence. I think we also have to be fair to trainees-students and residents-to make sure that our expectations are clear and that trainees are given feedback during the process. Finally, we have to be fair to teachers as well and I'll address each of these a little bit during this talk. Part of the idea of fairness to society is to ensure that there is validity to the process of evaluation, and we'll talk about both content validity, assurance that you're looking at what you want to look at, and predictive validity, that measurements during training actually provide some prediction of whether or not you're competent later. Reliability is also an important part of fairness to students, because it means assessment is consistent between teachers, between different sites, and from year to year. If you're a resident in surgery at Walter Reed, the evaluation should be consistent with that at Bethesda or Brooke or anywhere else. Finally, feasibility is important in evaluation. It has to be timely, portable, and cost-effective. These are the big characteristics of evaluation: fairness, validity, reliability, and feasibility. I am now going to focus on how we can accomplish this.

I will emphasize definitions and distinctions, such as differences between evaluation and assessment, between dichotomous and scalar judgments, and between analytic and synthetic methods of evaluation. I will ask the traditional, "cardinal" questions of who, what, when, where, why, and how, that I believe anyone involved in training should be able to answer for their own program and that the different working groups in this conference have to address in some detail over the next few days.

The first distinctions I want to make have to do with the phases in evaluation, and I will use the word assessment to mean making an observation about a trainee. The word assessment is rooted in the French word assay, like a radioimmunoassay. It has to do with measurement. It is expressed in numbers and it has to do with the tools that we use to measure. I will use the word evaluation to mean "determining value"-the root of the word evaluation is value. This is something that is expressed in words, not numbers, having to do with the expected level for a given trainee. This judgment is influenced by context, whether one is assessing a student, a resident or someone actually in practice. In a sense, evaluation is an educational diagnosis that someone has achieved and demonstrated the values that the professional community expects for their level of training. The third and final phase has to do with grading or feedback. These are actions, not judgments. Feedback is an educational action and grading is an administrative or social action. In summary, the process of measuring success includes assessment, evaluation, and then grading, or feedback.

I like to use clinical examples, and I would like to suggest an analogy to a thyroid patient. If we obtain a thyroid-stimulating hormone value, we get a measurement, for example, 0.7 mU/L. This is simply a measurement and it does not tell us anything more than what the result is. To evaluate or interpret it, we have to place it in context. If, for instance, this patient had a free T-4 that was very low, then this thyroid-stimulating hormone level is clearly inappropriate and our evaluation, or conclusion, would differ accordingly. We would take actions-a diagnostic action, like a magnetic resonance image of the pituitary, and an administrative action, such as a medical board. This would represent the response to the conclusion or feedback. Now let me draw another analogy to a medical situation. If our educational assay is clinical simulation, with an actor or "standardized patient," of smallpox in an emergency room patient, we will set performance standards that we think trainees should meet and these standards will allow our assessment. This will generate a rating, score, or number. We then have to place the assessment in context, that is, to evaluate, to determine the value, and we may have different standards for a student than we do for a finishing resident. We may expect the student to recognize a problem, but expect the resident to be able to manage it. If a resident only recognizes the problem, but not more than this, the resident would not be judged as competent, and our action might be to have them repeat the curriculum, or we may deny certification in military medical expertise. In summary, our three phases represent observation, reflection, and action. This is the rhythm of all productive human activity.