Basic statistics and the inconsistency of multiple comparison procedures

Canadian Journal of Experimental Psychology, Sep 2003 by Saville, David J

For estimating the variability in the estimated difference between any two experimental treatments, a single estimated standard error of the difference (SED) between the two treatment means can be calculated using the formula

This SED applies equally to all pairs of experimental treatments (Figure 1).

The 95% CI associated with the difference between any two of the treatment means is

(difference in means) or - x (t critical value)

using the t critical value described above (Figure 1). For example, if the residual degrees of freedom is 16, the 95% CI is [(difference in means) or - SEM x [the square root of]2 x 2.120]; this reduces to [(difference in means) or - 3 x SEM] in this cunningly chosen example! Again, the confidence interval has the same width for each pair of treatments.

To determine whether one treatment mean differs "significantly" from a second treatment mean, one approach is to determine whether zero is included in the 95% CI for the difference between the two treatment means. If zero is included, then "no difference" is a plausible scenario, so the difference between the two treatment means is "not significant." This is equivalent to determining whether the difference between the two means is less than SED x (t critical value), the half-width of the 95% CI; this latter quantity is therefore called the "Least Significant Difference"

Of course, LSD(1%) and LSD(0.1%) values can be calculated if the experimenter wishes to determine whether the difference is significant at p

As an aside, note that the pooling of the standard deviations between experimental treatments simply means that the pooled standard deviation is a more accurate estimate of the "true" common standard deviation than could be obtained from each treatment individually, and hence the resulting LSD is also a more accurate estimate of its "true" value. Therefore the effect of increasing the number of experimental treatments is simply to increase the accuracy of estimation of the LSD, not to decrease or increase the estimate.

The above procedure for determining which pairs of treatment means differ significantly is an example of a multiple comparison procedure. The one described above is called the unrestricted (or unprotected) LSD procedure (Saville, 1990). This procedure is equivalent to carrying out multiple t-tests of the form

t = (difference between two means) / SED

subject to the restrictions that the SED is based upon a pooled variance estimate as described above, and the t critical value has the corresponding pooled residual d.t. It is also equivalent to carrying out multiple F tests of the form

F = (difference between two means)^sup 2^ / (SED)^sup 2^

subject to the same restrictions. Some of the pros and cons of such a procedure will be discussed in the next section of this paper.

From the above discussion, it will be apparent that if an author reports any one of the statistical measures MSE (s^sup 2^), SEM, SED or LSD(5%), the other measures can be calculated by the reader, assuming that the methods section of the paper gives the sample size (n) and enough information on the statistical design for the reader to calculate the residual degrees of freedom. However, some measures are clearly more convenient for the reader than other measures.