Exploring the limits of science and beyond: research strategy and status

Journal of Parapsychology, The, March, 1994 by J.E. Kennedy

Meta-analyses of ganzfeld experiments also appear to support the goal-oriented experimenter-effects model. In a review of 42 ganzfeld studies, Hyman (1985) found that the significance levels did not follow the relationship with sample size that is normally expected in statistical research. The studies with small sample sizes were more significant than would be expected and the difference between the observed and expected values was statistically significant (Honorton, 1985; Hyman, 1985). Hyman speculated that this "strange" result indicated selective reporting of significant results. However, an investigation of selective reporting failed to support this hypothesis (Blackmore, 1980). Of course, this strange result is expected under the goal-oriented experimenter-effects hypothesis.

A further series of 11 ganzfeld studies gave similar results. Bem and Honorton (1994) reported a significant negative correlation between effect size and sample size. Because effect size is basically the z score divided by the square root of the sample size, this result is consistent with the goal-oriented psi hypothesis and is significantly different from the usual assumptions for statistical research.(4) These studies were free of any selective reporting bias. Also, the negative correlation was not due to decline effects interacting with the varying lengths of the studies because these studies did not have systematic declines in hits across sessions.

A meta-analysis of early ESP experiments was also consistent with goal-oriented experimenter effects but the results were confounded. In a meta-analysis of the ESP studies listed in Rhine et al. (1940), Nash (1989) correlated the sample size and scoring rate for each different a priori probability of a hit. The scoring rate is a measure of effect size for a given a priori probability of a hit. The correlations were negative for each of the 13 different levels of a priori probability (p = .0002). Although this result is consistent with the goal-oriented experimenter-effects hypothesis, it is confounded because sample size was also correlated with number of trials per subject and quality of methodology, both of which could affect scoring rate.

Although meta-analysis support for the goal-oriented psi hypothesis seems remarkably consistent, it must be taken with caution until several technical issues are addressed. They include the following: (a) the lines in Figure 2 vary with psi strength, and psi strength appears to vary among experimenters (Kennedy & Taddonio, 1976; White, 1976b); (b) the slopes in Figure 2A may vary with the a priori probability of a hit; (c) selective reporting of significant results could bias and hide the low end of the lines in Figure 2A; (d) both the sample size and the quality of methodology often increase as a line of research evolves, and the scoring rate may decrease as the methodology improves; (e) decline effects combined with differing numbers of trials per subject could give differing scoring rates; and (f) the relevant goal for the psi source may vary with researchers or experimental procedure. These factors could lead to misleading results on meta-analyses evaluating goal-oriented experimenter effects. Methods for addressing these factors include analyzing selected subsets of experiments and/or carrying out sensitivity analysis (probably simulations) to explore how these factors affect the results.