A house with no foundation

Issues in Science and Technology, Fall 2003 by Risinger, D Michael, Saks, Michael J

Forensic science needs to build a base of rigorous research to establish its reliability.

Many of the forensic techniques used in courtroom proceedings, such as hair analysis, fingerprinting, the polygraph, and ballistics, rest on a foundation of very weak science, and virtually no rigorous research to strengthen this foundation is being done. Instead, we have a growing body of unreliable research funded by law enforcement agencies with a strong interest in promoting the validity of these techniques. This forensic "science" differs significantly from what most of us consider science to be.

In the normal practice of science, it is hoped that professional acculturation reduces these worries to a functional minimum. To this degree, science is based on trust, albeit a trust that is defensible as reasonably warranted in most contexts. Nothing undermines the conditions supporting this normal trust like partisanship. This is not to say that partisanship does not exist in some form in most or all of the practice of science by humans, even if it is limited to overvaluing the importance of one's own research agenda in the grand scheme of things. But science is a group activity whose individual outputs are the product of human hopes and expectations operating within a social system that has evolved to emphasize the testing of ideas and aspirations against an assumed substrate of objective external empirical fact.

The demands of the culture of science-ranging from the mental discipline and methodological requirements that constitute an important part of the scientific method to the various processes by which scientific work is reviewed, critiqued, and replicated (or not)-tend to keep human motivation-produced threats to validity within acceptable bounds in individuals; and the broad group nature of science ensures that, through the bias cancellation that results from multiple evaluation, something like progress can emerge in the long run. However, in contexts where partisanship is elevated and work is insulated from the normal systems of the science culture for checking and canceling bias, then the reasons to trust on which science depends are undermined. Nowhere is this more likely to be a serious problem than in a litigation-driven research setting, because virtually no human activity short of armed conflict or dogmatic religious controversy is more partisan than litigation. In litigation-driven situations, few participating experts can resist the urge to help their side win, even at the expense of the usual norms of scientific practice. Consider something as simple as communication between researchers who are on different sides of litigation. Although there is no formal legal reason for it, many such researchers cease communicating about their differences except through and in consultation with counsel. What could be more unnatural for normal researchers? And what purpose does such behavior serve other than ensure that scientific differences are not resolved but exacerbated?

These concerns apply not only to research undertaken for use in a particular case, but also to research undertaken for use in unspecified cases to come, as long as the litigation interest of the sponsoring party is sufficiently clear. This is what differentiates litigation-driven research from much other interest-driven research. For instance, in research directed toward Food and Drug Administration (FDA) approval of drugs, drug companies are interested not only in positive findings but also in the discovery of dangers that might require costly compensation in the future or cause their drug to be less competitive in the marketplace. In other words, built-in incentives exist to find correct answers. In addition, the research will have to be conducted according to protocols set by the FDA, and it will be reviewed by a community of regulators who are technically competent and, at least in theory, appropriately skeptical. By contrast, in much litigation-driven research, there is a single unambiguous desired result, and the research findings will be presented to a reviewing community (judges and juries) that typically is not scientifically literate. These circumstances are more like the ground conditions relating to industry-sponsored research in regard to food supplements and tobacco, two areas of notoriously problematic claims.

Our attention is focused on an area that does not appear to figure prominently in most examinations of the problems of litigation-driven research: law enforcement-sponsored research relevant to the reliability of expert evidence in criminal cases, evidence that virtually always is proffered on behalf of the government's case. Of primary concern is research directly focused on the error rates of various currently accepted forensic identification processes, which have not been subject to any formal validity testing.

Illusion of infallibility

Many forces combine to raise special concerns in such areas. From the perspective of prosecution and law enforcement, any such research can result only in a net loss because in these areas there has been a carefully fostered public perception of near-infallibility. Research revealing almost any error rate under common real-world conditions undermines the aura. In addition, data that can show deficiencies in individual practitioners threaten that individual's continued usefulness as an effective witness. The combined effects of these two kinds of findings can potentially result in increased numbers of acquittals in cases where other evidence of a defendant's guilt is weak. Valid or not, however, such testimony is extremely useful to a prosecutor who is personally convinced of the guilt of the defendant (which, given the partisan nature of the litigation process, is virtually every prosecutor) and is willing to use whatever the law allows in an effort to convince the jury of the same thing. Consequently, research results calling into question the validity of such expertise, or defining its error rates, is threatening because it undermines a powerful tool for obtaining convictions and also threatens the status and livelihood of the law enforcement team members who practice the putative expertise.