Using planned comparisons in management research: a case for the Bonferroni procedure

Journal of Management, Fall, 1993 by Maria B. Castaneda, Joel R. Levin, Randall B. Dunham

Research in management often involves testing for differences in dependent variable levels across two or more identifiable groups of study participants. Such tests can be quite straightforward, as is the case when only two groups are involved in a study and the researcher's hypothesis involves a simple comparison (or "contrast") of these two groups. Other research designs, however, often present a more complex situation in which the researcher wishes to conduct multiple comparisons in the context of either a K |is greater than~ 2 one-way layout or a factorial design. As will be shown in this paper, the fact that a posteriori techniques are commonly used when a priori analyses are indicated results in a less-than-optimal test of the researcher's hypotheses. In addition, it is argued in this paper that the a posteriori ("post hoc") approaches commonly used in TABULAR DATA OMITTED such situations are less precise--and often less powerful--than is an extremely versatile a priori ("planned") alternative. This argument is made both conceptually and statistically and is illustrated through a detailed example.

Before we proceed, it might be of interest to examine current multiple comparison practices. Table 1 summarizes the use of various multiple-comparison practices in the field of management and applied research from 1985 to 1988. Three major journals were covered. These were the Journal of Management, the Academy of Management Journal and the Journal of Applied Psychology. As can be seen in this table, of the 745 published studies, 199 (27 percent) involved comparisons among means. Despite the fact that most of these studies postulated a priori hypotheses, only 49 of the 199 (25 percent) used any type of a priori statistical technique. The most common technique (46 out of 49) was a t test based on either an unknown or uncontrolled Type I error probability.

Although the just-presented survey considered studies only through 1988, there is evidence that similar "inconsistent" statistical practices have continued. A more recent survey of all articles published in the Journal of Management, TABULAR DATA OMITTED the Academy of Management Journal and the Journal of Applied Psychology during 1992 revealed that of the 160 published articles, 38 (24 percent) involved comparisons among means. A careful analysis of the designs of these studies clearly shows that most of them did postulate a priori hypotheses and yet only 7 of the 38 (18 percent) used any type of apriori statistical technique. As in the previous survey, the procedure of choice was an uncontrolled t test (6 out of 7).

Table 3 presents examples of representative statements from articles published in the Journal of Applied Psychology during 1992. Statement 1 shows example a study which relied on uncontrolled a priori t tests. Statements 2 and 3 show examples of studies that conducted "planned comparisons" after omnibus F tests. Statements 4, 5 and show examples of studies where the authors postulated a priori hypotheses but relied on a posteriori statistical techniques. As will be shown in this paper, the common practice of conducting "planned comparisons" after conducting an omnibus F test represents an "incoherent" use of a priori statistical techniques. Further, using post hoc comparisons to test a priori hypotheses results in less-than-optimal tests of the researcher's hypotheses.(1)

TABULAR DATA OMITTED

Simultaneous Multiple-Comparison Procedures

A linear combination of K means is called a "contrast" in means when its coefficients, |C.sub.k~, sum to zero. That is:

|Mathematical Expression Omitted~

is a contrast if and only if

|summation of~ |C.sub.k~ = 0 where k=1 to K.

In less formal terms, a contrast may be regarded as a difference involving two or more cell means.

Consider, for example, the situation in which an experiment has been conducted to examine the effects of an organizational intervention. Tests of the intervention effect are usually based on a factorial analysis-of-variance (ANOVA) design. In a two-way layout, traditionally one first conducts omnibus F tests of the two main effects and the interaction. If significant effects are detected using these omnibus tests, a post hoc multiple-comparison procedure (such as the one proposed by Fisher, Duncan, Newman and Keuls, Tukey, cheffe) is followed to examine specific between-level differences of interest.

The Scheffe (1953) Multiple-Comparison Procedure

As Kirk (1982, p. 121) has pointed out, once an overall F statistic is found to be significant, "Scheffe's procedure can be used to evaluate all a posteriori contrasts among means.... "An important property of the Scheffe procedure is that it is the only multiple-comparison procedure that is entirely "coherent" (Gabriel, 1969) with the omnibus F test. By that is meant that statistical decisions stemming from the F test and those stemming from Scheffe's procedure will always be in agreement. A significant omnibus F guarantees that one can find at least one significant Scheffe contrast--though not necessarily a substantively interesting, or even an interpretable, one--based on the same controlled Type I error probability (discussed shortly), whereas a nonsignificant omnibus F guarantees that all Scheffe contrasts are nonsignificant. No other multiple-comparison procedure can boast the same kind of coherence with the F test. In that sense, the analysis-of-variance F test is a logically consistent "screening device" for the Scheffe multiple-comparison procedure (and only for that procedure).(2)