Answering family physicians' clinical questions using electronic medical databases - Original Research

Journal of Family Practice, Nov, 2001 by Brian S. Alper, James J. Stevermer, David S. White, Bernard G. Ewigman

Scoring and Arbitration

The 2 physician searchers judged the adequacy of the answers to each question for each database. If the searchers both found adequate answers, the result was accepted as adequate, and the average time required to find and interpret the answer was recorded. If neither searcher found an adequate answer, then the answer was deemed inadequate. If only one searcher found an adequate answer, the second searcher evaluated that answer. If the answer was acceptable to the second searcher, it was considered an adequate answer, and the time for the first searcher was recorded.

When searchers disagreed on the adequacy of identified answers, an arbitration panel consisting of 3 family physicians who were not affiliated with any of the databases met' independently from the searchers to determine the adequacy of the answers by consensus.

Analysis

Our primary outcome was the proportion of questions adequately answered by each database. We calculated 95% confidence limits for the proportions of adequate answers. (7) Means and medians were determined for the time to reach adequate answers for each database. We calculated the [kappa] statistic for the independent findings of the 2 searchers and for the results after the searchers reviewed each other's searches. (8) We combined the results of individual databases to determine the proportion of questions answered by all combinations of 2, 3, and 4 databases. We considered the question adequately answered if any of the individual databases adequately answered the question.

RESULTS

Thirty-eight databases were nominated, and 24 did not meet our inclusion criteria. * Fourteen databases met the inclusion criteria (Table 3) and were evaluated with the set of 20 questions (280 answer assessments) by 2 searchers. The Figure summarizes the process of evaluating the answers. The initial agreement between searchers was good [kappa]=0.69). Discussion between the searchers resolved 21 (52.5%) of the 40 discrepant answer assessments. These were due to inadequate searching or timing out (searching for 10 minutes) by one searcher, who agreed with the adequacy of the answer found by the other searcher. The agreement between searchers at this stage was excellent ([kappa]= 0.94).

The remaining 19 discrepant assessments (for which the searchers had different opinions regarding the adequacy of the answers identified) were referred to the arbitration panel for determination of the final results. Ten of these were deemed adequate.

Results for individual databases in rank order of proportion of questions answered followed by average time to identify adequate answers are reported in Table 3. The combination of STAT!Ref and MDConsult could answer 85% of our set of 20 questions. Four combinations of 2 databases (STAT!Ref and either MAXX, MDChoice.com, Primary Care Guidelines, or Medscape) could answer 80% of our questions. Two combinations of 3 databases (STAT!Ref, MDConsult, and either DynaMed or MAXX) could answer 90% of our questions. Combinations of 4 databases answered the most sample questions (95%, 19/20). These combinations consisted of STAT!Ref, DynaMed, MAXX, and either MDConsult or American Family Physician.