Demonstration research and meta-analysis in parapsychology

Journal of Parapsychology, The, Sept, 1993 by Stanley Krippner, William Braud, Irvin L. Child, John Palmer, K. Ramakrishna Rao, Marilyn Schlitz, Rhea A. White, Jessica Utts

This is the second of several reports to be made available by the Parapsychological Association (PA) for people outside of the association who are interested in its activities. The previous report (Parapsychological Association, 1989) described four research approaches in parapsychology: survey research, field research, demonstration research, and process research. This report will focus on demonstration research, which involves the comparison of observed events with what would ordinarily be expected in a single carefully defined situation. In this type of research, statistically unlikely outcomes in parapsychological experiments indicate that events have occurred that do not appear to be explicable by known mechanisms and are unlikely to be coincidence, but the research does not give information about how or why the events occurred.

Demonstration research is research aimed at obtaining incontrovertible evidence for the existence of a phenomenon. It may mean conducting a crucial experiment that has appropriate controls against all possible sources of error. In such complex subjects as parapsychology, it is hardly feasible to prespecify all sources of possible error and hence to control against them; it is always possible to invent--after the fact--a scenario of possible error. In areas where the hypothesis of experimenter fraud is entertained, there can be no crucial experiment inasmuch as the fraud hypothesis, in the final analysis, is itself unfalsifiable. Therefore, parapsychologists generally tend to favor replication as a means of demonstrating psi instead of conducting the so-called crucial experiment.

The Issue of Repeatability

The literature on demonstration research in parapsychology is embedded in the assumption that the field needs to develop a repeatable experiment in order to establish itself as a science. PA members have diverse views on this topic; some agree with critics of the field that strict repeatability deserves a high priority in the field. Most, however, argue that strict replication is not essential; instead, they require the kind of statistical replicability that is common in most of the social and behavioral sciences (see Shapin & Coly, 1985). There are also those who take the extreme position that nonreplicability is inherent in the very nature of psi and how it functions in nature (e.g., Eisenbud, 1983). It is generally agreed that the attention given to this issue should not eclipse other important evaluation criteria such as the control of sensory channels and the elimination of alternative (i.e., non-psi) explanations in appraising data.

Parapsychology, perhaps because many of its leaders in the past three decades were trained as psychologists, has evidenced a strong tendency to model itself after experimental psychology. In this paradigm, the experimental approach is the method of choice and repeatability of experimental results is considered to be a primary aim. Critics of parapsychology tend to concur on the value of this approach, but they have not made it clear just what would constitute acceptable evidence of a repeatable experiment. In the early days of psi research, some of these critics insisted that a single "fraud-proof" experiment would convince them of the reality of ESP. However, as soon as well-designed experiments showing statistical significance emerged, critics realized that a single experiment could be significant simply by chance. Critics then emphasized the need for repeatability, and one critic suggested that a significant experiment be replicated at least two times to demonstrate that the result was not due to chance (Hansel, 1980, p. 298). Several experimental approaches, usually those comparing two attitudinal types (e.g., "believers" in psi and "non-believers," introverts and extraverts) have yielded results that achieved a striking level of replication (Utts, 1991).

Parapsychologists, again following psychology, were able to detect another problem with this line of reasoning. Replication alone could not establish a result if no consideration is given to sample size or power; only an experiment with extremely high power should be expected to yield consistent results three times in succession, a goal for many parapsychologists. A more appropriate analysis would compare the effect sizes for similar experiments across experimenters and across time to see if there have been consistent effects of similar magnitude (Utts, 1991). According to Rosenthal (1990), "the traditional view of replication focuses on significance level as the relevant summary statistic of a study and evaluates the success of a replication in a dichotomous fashion. The newer, more useful view of replication focuses on effect size as the more important summary statistic of a study and evaluates the success of a replication not in a dichotomous but in a continuous fashion".

The Role of Meta-Analysis

This broader approach to replication is reflected in the development of "meta-analysis," a label coined by Glass (1976) that has come to refer to a variety of strategies for quantitatively summarizing evidence across entire areas of research. In principle, meta-analytic summaries must adhere to the same logical requirements associated with adequate summaries of evidence within studies. However, that task is necessarily more challenging: An adequate meta-analytic summary presumes (a) that evidence within each study has been summarized adequately, and (b) that these summaries can themselves be summarized adequately (Hunter & Schmidt, 1990).