Measuring outcomes for military medical education

Military Medicine, Sep 2003 by Jackson, Jeffrey L, O'Malley, Patrick G, Hemmer, Paul, Inouye, Lisa, Et al

Background

Assessment of the quality of medical care has traditionally occurred within three domains: structure, process, and outcomes. The same model for assessing the quality of military medicine and, specifically, medical education for trainees in military medicine should also apply. This section will describe the principles and challenges of measuring quality in military medicine.

Principles of Quality Measurement

Historically, quality assessment efforts have focused primarily upon structure and process rather than outcomes for several reasons. First, it is generally believed that appropriate structures and processes for medical care must be in place to achieve desirable outcomes, although this is an unproven assumption. Second, structure and process is under the direct control of the health care system; while these constructs may impact on patient outcomes, there are likely a host of other factors that may be even more important in shaping outcomes. A third rationale for emphasizing structure and process is that they are comparatively easy to measure. Checklists can be used to quickly and efficiently evaluate structure. Processes pose a somewhat greater challenge, but surrogate markers such as standard operating procedures or medical charts can be reviewed in lieu of directly observing care processes. Consider the following example for reviewing lumbar punctures. One could review the standard operating procedure that details departmental policy on how lumbar punctures are to be performed, pull the medical charts of individuals who have undergone lumbar puncture to see whether essential elements are documented (e.g., informed consent, immediate complications, etc.) and whether a lumbar puncture is performed within the window of review, and one could observe a lumbar puncture being performed. The validity of a process measure as a surrogate marker of quality is dependent upon how well variation in that process measure has been demonstrated to be associated with variation in the outcome of interest. By and large, measuring outcomes is a greater challenge.

Two important challenges in outcome assessment are risk adjustment and attribution. Risk adjustment refers to adjusting outcome data to reflect different patient populations or variation in other variables that might affect the outcomes. For example, a particularly gifted cardiovascular surgeon may very well wind up doing all of the complicated cases in the region, cases other surgeons avoid. His or her surgical outcomes, examining only morbidity and mortality, may appear to be much worse than a colleague who performs solely low-risk cardiovascular procedures. Without risk adjustment, the data may be subject to abject misinterpretation. A military medicine education example might be that when looking at the quality of a military medicine education program as measured by a military medicine outcome (e.g., battlefield injury rates, battlefield mortality), one would want to adjust for type of battle, predominant weapon of battle, comorbid epidemics, battlefield climate, etc. as well as other military relevant variables that are associated with battlefield health outcomes.

Unfortunately, approaches for risk adjustment remain controversial and difficult. Proper adjustment depends on capturing the proper "adjusting" variables. Such tools have yet to be fully developed. Attribution of outcomes can be difficult. Any particular outcome may depend on a host of factors, many of which may not be in the control of the particular clinician. If a diabetic patient dies prematurely as a consequence of stepping in front of a bus, does that count as a poor outcome of his care? Perhaps it should if it was due to overly low blood sugars with mental confusion. If a soldier is killed by friendly fire should that be considered a bad outcome attributable to the field or brigade surgeon? Again, perhaps it should if the Medical Corps officer was not involved enough in participating in the military decision making (on both ends). A final barrier to outcomes is that they are costly to measure. Long-term follow-up of large numbers of patients is often necessary.

Military Medicine Education Quality: Structure and Process Prior to considering measurement of outcomes in response to modifications in military medical education, the group felt that the first order of business was to ensure that the structure and processes of military medical care are in good order. Currently, these are often fragmented and insufficiently documented, particularly with regard to the educational component. Uniformed Services University of the Health Sciences graduates receive extensive militarily relevant medical training, and some studies suggest that they are better prepared to assume operational roles upon completion of residency, (references?) but clinicians entering the military from Reserve Officer Training Corps or Health Professions Scholarship Program or through direct accession may have had little or no militarily relevant training. Once on active duty, militarily relevant training during residency is at the discretion of each individual training program and the content and range of experiences differs markedly. While specialty-specific Military Unique Curriculum documents have been published, most observers agree that they have been underutilized.1-3 There have been some successes. A military unique curriculum for Army interns across all medical centers and across all specialties has been successfully implemented. Such efforts need to continue to progress across all residency training programs in all branches of military service to ensure that residents are able to assume appropriate operational roles after completing their training. There are several different possible models for such educational dissemination. The Army intern curriculum is based on a series of Powerpoint lectures provided by faculty from all of the Army medical centers, at a common web site. Another alternative could be a cadre of instructors that travel from medical center to medical center, although this could be an expensive option. Another possibility would be to develop a curriculum centrally, and "train the trainer," i.e., bring faculty from each center to a central location to be taught how to teach the curriculum and return to their own institution to train faculty in the curriculum locally.4 Militarily relevant instruction has been an element of a faculty development course for military program and clerkship directors for several years.5