Outcomes Monitoring and the Testing of New Psychiatric Treatments: Work Therapy in the Treatment of Chronic Post-Traumatic Stress Disorder

Health Services Research, April, 2000 by Robert Rosenheck, Marilyn Stolar, Alan Fontana

Objective. To evaluate the effectiveness of a work therapy intervention, the Department of Veterans Affairs (VA) Compensated Work Therapy program (CWT), in the treatment of patients suffering from chronic war-related post-traumatic stress disorder (PTSD); and to demonstrate methods for using outcomes monitoring data to screen previously untested treatments.

Data Sources/Study Setting. Baseline and four-month follow-up questionnaires administered to 3,076 veterans treated in 52 specialized VA inpatient programs for treatment of PTSD at facilities that also had CWT programs. Altogether 78 (2.5 percent) of these patients participated in CWT during the four months after discharge.

Study Design. The study used a pre-post nonequivalent control group design.

Data Collection/Extraction Methods. Questionnaires documented PTSD symptoms, violent behavior, alcohol and drug use, employment status, and medical status at the time of program entry and four-months after discharge from the hospital to the community. Administrative databases were used to identify participants in the CWT program. Propensity scores were used to match CWT participants and other patients, and hierarchical linear modeling was used to evaluate differences in outcomes between treatment groups on seven outcomes.

Principal Findings. The propensity scaling method created groups that were not significantly different on any measure. No greater improvement was observed among CWT participants than among other patients on any of seven outcome measures.

Conclusions. Substantively this study suggests that work therapy, as currently practiced in VA, is not an effective intervention, at least in the short term, for chronic, war-related PTSD. Methodologically it illustrates the use of outcomes monitoring data to screen previously untested treatments and the use of propensity scoring and hierarchical linear modeling to adjust for selection biases in observational studies.

Key Words. Outcomes, quality, post-traumatic stress disorder, propensity scores

OUTCOMES MANAGEMENT AND TECHNOLOGY ASSESSMENT

Over a decade ago, outcomes management in healthcare was described (Ellwood 1988) and heralded as the "third revolution" in twentieth-century medical care (Relman 1988). Since that time, it has been widely accepted that quality in healthcare cannot be adequately assessed by traditional indicators such as professional licensure, continuing medical education, focused chart reviews, or even adherence to recommended clinical pathways (Brennan and Berwick 1996; Epstein 1990). Meaningful assessment of the value of healthcare services requires, in addition, the evaluation of clinical outcomes assessed by direct, systematic measurement of the results of treatment (Blumenthal 1996; Millenson 1997; Sederer, Dickey, and Hermann 1996).

In one of the earliest explications of the outcomes management imperative, Ellwood (1988:1552) stated that "outcomes management's closest relative is the clinical trial." He suggested that outcomes monitoring data would eventually be used to compare existing treatments and to evaluate new technologies, thereby avoiding both the expense of costly clinical trials and the loss of generalizability that was often entailed in selective recruitment for such trials. Ellwood envisioned the construction of comprehensive national outcome databases for specific diseases and the use of those databases to test promising treatments. Such treatments could be tried initially on a small number of patients. These patients could be matched with similar patients in a large database to allow assessment of the relative effectiveness of the new treatment.

Although outcomes monitoring has been embraced as the desired standard for quality assessment, the practical difficulties and costs of implementing such extensive data collection have severely limited its deployment in actual practice (Steinwachs, Wu, and Skinner 1994; Clardy, Booth, Smith, et al. 1998). Furthermore, Ellwood and others did not address the problem of potentially confounding differences in the types of patients who receive different treatments in general practice--the very reason why the randomized clinical trial remains the gold standard for evaluating clinical efficacy. Thus, even though the use of outcomes monitoring data to test new treatments is promising in theory, we are aware of no previous studies in which this approach has been applied in the field of behavioral healthcare.

WAR-RELATED POST-TRAUMATIC STRESS DISORDER (PTSD)

The treatment of war-related post-traumatic stress disorder (PTSD) is a major priority for the Department of Veterans Affairs (VA) healthcare system. National survey data indicate that almost 500,000 veterans of the Vietnam era meet minimal diagnostic criteria for PTSD (Kulka, Schlenger, Fairbank, et al. 1990), and 80,000 veterans suffering from debilitating problems such as nightmares, flashbacks, and profound social withdrawal seek help for war-related PTSD from VA each year (Rosenheck et al. 1997). Outcome studies suggest that conventional psychosocial and pharmacotherapeutic treatments have limited efficacy, especially in severe and persistent cases of PTSD (Fontana and Rosenheck 1997a; Rosenheck and Fontana 1996; Solomon, Gerrity, and Muff 1992; Davidson 1997). It is now over two decades since the last U.S. soldier left Vietnam and the psychiatric sequelae of the Vietnam war are, by definition, chronic and, in most cases that present to the VA, seriously disabling (Rosenheck and Fontana 1996).