Basic statistics and the inconsistency of multiple comparison procedures

Canadian Journal of Experimental Psychology, Sep 2003 by Saville, David J

Abstract This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD(5%)). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several wellknown MCPs by focussing on a particular operating characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments. The procedure that behaves best according to this criterion is the unrestricted least significant difference (LSD) procedure. The unrestricted LSD is therefore recommended with the proviso that it be used as a method of generating hypotheses to be tested in subsequent experimentation, not as a method that attempts to simultaneously formulate and test hypotheses.

The format for this paper is as follows. First I shall discuss the various ways in which statistical information can be summarized in scientific papers, and recommend usage of the least significant difference (LSD) as a succinct and useful measure of experimental variability. Then I shall illustrate differences in an important operating characteristic between various multiple comparison procedures by analyzing a specific data set from an experiment with 32 treatments, and subsets corresponding to selected treatments. This leads to a recommendation that planned and unplanned comparisons between treatment means should be made using the same statistical tool (e.g., a 5% level test per comparison), without any adjustment for multiplicity. The proviso is that experimenters should clearly distinguish between the formulation of new hypotheses involving such comparisons (that require confirmation in subsequent experimentation), and the testing of pre-existing, a priori hypotheses that are being confirmed in the current experiment.

Basic Statistical Measures

As an agricultural research statistician attempting to write an article for an unfamiliar readership, I decided to browse through the 2001 volume (No. 55) of the Canadian Journal of Experimental Psychology in order to gain an appreciation of the usual method of presenting statistical information. I especially searched for papers in which data were assumed to follow a "normal" distribution, leading to the calculation of pooled or unpooled standard deviations and related statistical measures. I selected five such papers for copying and further perusal (Baranski & Petrusic, 2001; Christie & Klein, 2001; Gold & Pratt, 2001; Hubbard, 2001; Shore, McLaughlin, & Klein, 2001).

My study of these five papers yielded the following. In all five papers, F values, MSE (Mean Square Error) values and /; values were reported; these summarized the output produced by the analysis of variance technique. In two of the five papers, the SEM (Standard Error of Mean) was presented, in another paper, 95% confidence intervals were presented for each mean, in another paper the PLSD (Protected Least Significant Difference) was presented, and in the fifth paper none of these three measures was presented. I shall now discuss these alternative measures, spell out the links between them, and describe their relationship to the analysis of variance.

When the analysis of variance technique is employed for data analysis, the implicit assumption is that all treatments have similar standard deviations. This assumption is routinely checked by plotting the residuals (or errors) from the model against the fitted values; if the resulting scattergram forms a "

Under this assumption that all experimental treatments have the same standard deviation (or are homogeneous in variance to use statistical jargon), a single estimated standard errorof the mean (SEM) can be calculated using the formula

if all experimental treatments have the same sample size n (Figure 1). This SEM applies equally to all of the experimental treatments.

The 95% confidence interval (CI) associated with each of the treatment means is

mean + or - SEM x (t critical value)

where the t critical value is the 97.5 percentile of the t distribution with the residual (or pooled error) degrees of freedom associated with the analysis of variance model (Figure 1). For example, if the residual degrees of freedom is 16, the 95% CI is [mean + or - SEM x 2.120]. The confidence interval has the same width for each treatment (under the assumptions of the analysis of variance).