A proposal and challenge for proponents and skeptics of psi

Journal of Parapsychology, The, Spring, 2004 by J.E. Kennedy

It appears to me that preplanned analysis of studies with sufficient sample size to reliably obtain significant results is necessary to provide convincing experimental results and meaningful probability values in controversial settings such as parapsychology. Sample sizes should be set so that the probability of obtaining significant results is at least .8 given a realistic psi effect. This is a substantial change from the current practice in which studies are done with little regard for statistical power and only about 20% to 30% are significant, which results in controversy and speculation about whether the predominately negative results are due to a lack of psi or a lack of sample size. Performing a prospective power analysis is simply doing what statisticians have long recommended.

If the claims that meta-analyses results provide evidence for psi are actually valid, then this approach of prospective power analysis and study planning will be successful. If this approach will not work, then the application of statistical methods in parapsychology, including meta-analyses, will not be convincing.

From my perspective now, it would make good sense to form a committee consisting of experienced parapsychologists, moderate skeptics, and at least one statistician to review and comment on protocols for pivotal experiments prior to the experiments being carried out. The committee could also do independent analyses of data, verify that the analyses comply with those planned in the protocol, and perhaps sometimes do site inspections. The possibility of a detailed, on-site, critical audit of the experimental procedure and results provides a healthy perspective on methodology. It would be valuable to have this option available even if it is rarely or never used.

The idea of a registry to distinguish between exploratory and confirmatory experiments has been suggested several times over the years (e.g., Hyman & Honorton, 1986; also see comments in Schmeidler & Edge, 1999). This strategy would allow researchers more freedom to do exploratory studies as long as the confirmatory or pivotal studies are formally defined in advance.

The present proposal is an extension of the registry idea that would also attempt to resolve much of the methodological controversy before rather than after a study is carried out. The most efficient strategy to obtain a consensus is to have those people who are critical provide input and agree on the research plan from the beginning. The net effort to carry out a study and answer criticisms may actually be less, and the final quality of evidence should be substantially higher.

This strategy is consistent with the idea that only certain experimenters can be expected to obtain positive results and does not require that any experimenter, no matter how skeptical, must be able to consistently obtain significant results. Thus, this strategy is reasonably consistent with the known characteristics of psi research.

This strategy also allows starting with a clean slate for evaluating research. The studies that comply with this process can stand as a separate category to determine whether there is evidence for psi. Given the higher quality of each pivotal study, there would be less need for many replications, and experimenters would have more freedom to capitalize on the novelty effect of starting new studies. A pre-specified analysis and criteria could be set for determining whether a group of studies provides overall evidence for psi. This could focus on certain experimenters with a track record of success, rather than expecting any and all experimenters to be successful.