Effects of information sequence and irrelevant distractor information when using a computer-based decision aid

Decision Sciences, Winter 2001 by Reneau, J Hal, Blanthorne, Cindy

Effects of Information Sequence and Irrelevant Distractor Information When Using a Computer-based Decision Aid*

ABSTRACT

Computer-based decision aids are intended to support and improve human judgments. Frequently, the largest portion of the design effort is devoted to the technical aspects of the system; behavioral aspects are often overlooked. As a result, the decision aid may be ineffective. An experiment was conducted to examine the effects of two information structure variables that theoretically affect judgments: information sequence and irrelevant distractor information. Auditor subjects made continuing existence judgments for client-banks after interacting with one of four alternative decision aids. The decision aids are modifications of a system developed by an international CPA firm. Judgments were predicted to be more accurate when: (1) diagnostic information is presented late rather than early in the information sequence and (2) when no irrelevant distractor information is presented. Further, judgment confidence was predicted to be unrelated to either information sequence or irrelevant distractor information. The experimental data support all three predictions.

Subject Areas: Auditor Judgment, Experimental Design, and Human/Computer Interaction.

INTRODUCTION

Two consistent results emerge from numerous years of research conducted to examine various forms of decision aids. One is that unaided human predictions are less accurate than predictions derived from statistical models (see review by Kleinmuntz, 1990). The second is that humans will not likely allow their important predictions to be based solely on a statistical model (Ashton, 1990; Boatsman, Moeckel, & Pei, 1997; Brown & Jones, 1998; Kleinmuntz). Thus, it is important to better understand the factors affecting (1) deviations of human judgments from predictive models and (2) human reliance on predictions derived from statistical models. This study will focus on the former.

When constructing a decision aid the designer must decide what information will be presented and the form in which the information will be presented. These factors can influence the decision maker's judgments by influencing how the problem is structured and by affecting the manner in which the information is evaluated and combined into an overall judgment.

The purpose of this study is to examine the effects of two information system structure variables on human predictions that are aided by a computerized decision aid. The two variables examined, information sequence and irrelevant distractor information, are variables that have a theoretical relationship to judgment accuracy. Information sequence can be easily controlled when developing a computerized decision aid; and irrelevant distractor information, when identified, can be omitted from the decision aid.

The experimental setting is one where external auditors make continuing existence judgments for client financial institutions. This unique setting was chosen for several reasons. First, an actual computerized decision aid designed to support auditors' judgments of continuing existence was available, as were auditor subjects who were familiar with the microcomputer hardware used to implement the decision aid. In addition, the specific setting and subjects represent a harsh environment in which the usual benefits of a decision aid are least likely to be observed. That is, the environment auditors face in their day-to-day decisions contains financial incentives for performance and requires individuals to justify their judgments to others. These factors are thought to reduce the benefits derived from using a decision aid (Ashton, 1990). Further, this task has an observable criterion event which makes it possible to examine judgment accuracy rather than consensus.

The next two sections of this report briefly describe the institutional setting where auditors make continued existence judgments and the computerized decision aid examined in the experiment. The third section summarizes the previous research used to form the hypotheses of the study. Subsequent sections describe the method and results of the study, followed by some concluding comments.

INSTITUTIONAL SETTING

Hundreds of financial institutions in the United States experienced financial difficulty in recent years. The financial press reported many acquisitions and regulator closures of troubled financial institutions. External auditors play a non-trivial role in reporting on and, according to some, influencing the financial condition of these firms. Investigations by congressional committees into auditor involvement in the much publicized savings and loan failures support the importance of the auditor role (U.S. General Accounting Office, 1991). Auditors are required to assess the entity's "ability to continue as a going concern" (AICPA, 1988). If the auditor concludes there is a "substantial doubt" of the entity's continued existence, the audit report must be modified to include an explanatory paragraph reflecting that conclusion. This modification of the auditor's report provides an unambiguous signal of financial difficulty to the market and regulators.