A response to Storm and Ertel - 2002 - Brief Article

Journal of Parapsychology, The, June, 2002 by Julie Milton, Richard Wiseman

Many parapsychologists appear to be unaware of Storm and Ertel's (2001) recent meta-analysis of ganzfeld studies that appeared in psychological Bulletin last year subsequent to our own (Milton & Wiseman, 1999). To put their latest response into context, it is important to understand how their meta-analysis was conducted.

The standard method for retrieving studies for a review in an unbiased way is to search the literature systematically and apply consistent inclusion criteria to the studies retrieved. Storm and Ertel (2001), in contrast, performed a literature search for studies published during the mid-1980s but drew the rest of the studies they examined from meta-analyses conducted by Honorton (1985), Bem and Honorton (1990), and ourselves (Milton & Wiseman, 1999). These meta-analyses had different inclusion criteria from each other, with the result that Storm and Ertel excluded 14 studies that Honorton did not include in his 1974-1981 database because they did not report direct hits, but they included all 30 studies in our meta-analysis regardless of whether they reported direct hits. It makes no sense to assemble a database in this inconsistent way.

The early ganzfeld studies examined by Hyman (1985) and Honorton (1985) had methodological problems that were well documented in their articles. For example, Honorton (1985, p. 71) reported that only 36% of the studies used duplicate target sets of pictures to avoid handling cues. The problems in interpreting the studies' combined results in the face of such methodological problems led Hyman and Honorton (1986) to make their well-known joint statement "that the final verdict [concerning the evidence for psi] awaits the outcome of future experiments conducted by a broader range of investigators and according to more stringent standards" (pp. 352-353).

Nevertheless, Storm and Ertel's (2001) meta-analysis combines the results of the early ganzfeld studies with those of later studies. They devised a quality scale that excludes important methodological safeguards for ganzfeld studies, such as the use of duplicate target sets. They used the scale to weight the effect sizes of the 11 studies retrieved from their own literature search, while leaving the other 68 studies unassessed and unweighted, including the extensively criticised 1974-1981 studies. When producing a quantitative summary of a group of studies, it makes no sense to distort their effect sizes by applying quality weights, nor does it make sense to apply quality weights to some studies in a database but not to others. This practice is simply bizarre and cannot be the basis for making strong claims that the statistically significant cumulated outcome represents strong evidence for psi.

We pointed out these problems in our response to Storm and Ertel in Psychological Bulletin. Their reply in this journal does not indicate that there is any point in repeating our arguments, and it includes a number of maccurate descriptions of our statements and views. We refer readers to our original, detailed response (Milton & Wiseman, 2001). Meta-analysis may well have a future as a useful tool in parapsychology, but only if applied using justifiable methods.

(1.) For example, Storm and Ertel (2002) wrote, "Milton and Wiseman (2001) argued that the 11 pre-Communique studies used in our meta-analysis should not have been used at all because of ostensible 'methodological weaknesses"' (p. 76). This is not our view, and there is no such argument in our article. Storm and Ertel later stated, 'quality rating of the post-Communique studies would be redundant--Milton and Wiseman themselves regarded them as flawless by their own standards" (p. 77). We do not regard them as flawless and have made no statement to this effect.

REFERENCES

BEM, D.J., & HONORTON, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115, 4-18.

HONORTON, C. (1985). Meta-analysis of psi ganzfeld research: A response to Hyrnan. Journal of Parapsychology, 49, 51-91.

HYMAN, R. (1985). The ganzfeld psi experiment: A critical appraisal. Journal of Parapsychology, 49, 3-49.

HYMAN, R., & HONORTON, C. (1986). A joint communique: The psi ganzfeld controversy Journal of Parapsychology, 50, 350-364.

MILTON, J., & WISEMAN, R. (1999). Does psi exist? Lack of replication of an anomalous process of information transfer. Psychological Bulletin, 125, 387-391.

MILTON, J., & WISEMAN, R. (2001). Does psi exist? Reply to Storm and Ertel (2001). Psychological Bulletin, 127, 434-438.

STORM, L., & ERTEL, S. (2001). Does psi exist? Comments on Milton and Wiseman's (1999) meta-analysis of ganzfeld research. Psychological Bulletin, 127, 424-433.

STORM, L., & ERTEL, S. (2002). The ganzfeld debate continued: A response to Milton and Wiseman (2001). Journal of Parapsychology, 66, 73-82.

Department of Paediatric Epidemiology & Biostatistics

Institute of Child Health

University College London

30 Guilford Street

London WC1N 1EH

United Kingdom