Correspondence: the authors reply

Health Services Research, August, 1993 by Henry Krakauer, R. Clifton Bailey, Arthur J. Hartz, Alfred A. Rimm

Green and Wintfeld correctly point out one of the quandaries that arises when complex regression analyses are carried out, i.e., that the results are sometimes counterintuitive. Surprising values of regression coefficients may occur for two reasons. First, a real but unanticipated phenomenon may have been uncovered, but that is not likely the case here. Second, the variables with the surprising coefficients may be acting as surrogates for factors not included in the model or may be reporting a differential effect because their main physiologic effect was taken up by other variables explicitly present. These variables were left in the model for two reasons: (1) an extended and complex process of variable selection was employed to minimize the risk of misspecification; and (2) the major reason--the objective of the regressions was to produce sound estimates of the predicted probabilities of death. That we believe we have done. The predicted probabilities for hospitals in the highest 5 percent of the distribution differ by a factor of 3 from those in the lowest 5 percent (21.8 % vs. 7.2 %, Table 3). For predictions at the patient level, the clinical model produces a proportion of concordant pairs (area under the ROC curve) of 0.90. In addition, we tested the stability of the predictions by split-sample cross-validation, and were reassured by the results in the accompanying table.

TABULAR DATA OMITTED

(These results were not included in the article because of limitations of space.)

Green and Wintfeld also question whether either the claims-based or the clinical model account for any substantial component of the variation in mortality rates among hospitals, citing the high correlation between rankings of hospitals based on the observed mortality rates and on the claims model. This point is addressed directly, as we have noted above, by Table 3, which presents the distribution of observed rates (a fivefold range between the top and the bottom 5% of hospitals) and predicted rates (a threefold range for the clinical model and a twofold range for the claims model). Thus, a considerable portion of the range in observed hospital-specific mortality rates is predicted by the clinical model and, to a lesser extent, by the claims-based model. Indeed, in regressions of the observed mortality rates on the rates predicted by the clinical and the claims-based models, weighted by the number of cases at the hospitals, we find adjusted R-squareds of 0.71 and 0.39, respectively. (The unweighted R-squareds are 0.64 and 0.40, respectively.)

In all, we are satisfied that our analyses show that

1. The clinical model predicts patient outcomes well.

2. The clinical model produces stable predictions.

3. The clinical model accounts for a substantial proportion of the observed variation among hospitals in mortality rates.

4. The claims-based model gives results for hospitals that are similar in pattern to the clinical model, although they differ in certain specifics.

Our inference pertaining to the validity of the claims-based model was based in part on the rank order correlation of hospital regression coefficients, .91, obtained from analyses of the same population by means of that model and the clinical model. The slightly lower correlation coefficient of .88 cited by Wintfeld and Green was obtained in a comparison of the hospital regression coefficients, based on a population subset, with the residual mortality for the hospital derived from the published data, which employed a more complete population set. Our conclusion was that the claims-based model adequately described the overall patterns of variation in mortality rates and was useful for initial screening. We also pointed out its limitation in that it did not "positively identify outlier hospitals as providers of problematic care."

We do not claim to have achieved perfection in the model, which includes the clinical data abstracted by means of MedisGroups. As we state in the discussion, "The model taken as the reference standard is burdened with substantial deficiencies." In our opinion, however, it does give a better sense of the limitation and of the proper applications of the data that the HCFA releases in its annual publication of mortality rates associated with hospitalization. A very important by-product of the publication is the stimulus that it provides for the dialog such as that entered into by Green and Wintfeld, which cannot but improve the quality of the analytic procedures for the evaluation of the performance of medical practices and providers.

Henry Krakauer, M.D., Ph.D. Uniformed Services University

R. Clinton Bailey, Ph.D. Health Care Financing Administration

Arthur J. Hartz, M.D. Ph.D. Medical College of Wisconsin

Alfred A. Rimm, Ph.D. Case Western Reserve University