Belly up to the error bar

Natural History, Nov, 1998 by Neil de Grasse Tyson

An implicit goal of the scientific method is to minimize human bias--one of the great sources of experimental blunder.

This is the second part of a two-part essay. "Certain Uncertainties," Part I, appeared in last month's issue.

Formal accounts of the scientific method typically describe a hypothesis-posing, experiment-conducting activity. You might see words such as induction, deduction, cause, and effect. What's missing are the words creativity and uncertainty. Science can be a process in which practically anything goes--from middle-of-the-night hunches to mathematical formulations driven by classical aesthetics--so long as the results accurately describe and predict phenomena in the real world.

When conducting an experiment on the frontier of human understanding of tine universe, you never know what the right answer is supposed to be. Sometimes you don't even know the right question. Often, guided by a particular vision of how the universe works, all you can do is make a series of measurements that you hope will lead you to the right answer. Data gathered to answer such questions as How far away is the Moon? or What is the mass of the Sun? lend themselves to standard statistical analysis. But answers to other questions--such as What kind of cheese is tine Moon made of? do not, because the question starts with the false assumption that tine Moon is a cheesy place, which would most likely inhibit your acquisition of relevant data.

In most astrophysics experiments, some measurements will come out above the true value, while some will come out below. These are ordinary fluctuations: a chart of all the data points would look like the statistician's beloved bell curve. The history of science has shown that if an experiment is well designed, then most of the data will cluster around some value, presumably the right one.

Unfortunately, this "right" value may bear little correspondence to the real world when human bias is involved. An implicit goal of the scientific method is to minimize human bias, for therein lie some of the greatest sources of experimental blunder. When making multiple measurements, scientists occasionally discard values that deviate strongly from their expectations. Such selective editing can skew data and fatally compromise the experiment. Once results have been published, the experimenter may be the only one who knows which data were included and which discarded. (In all fairness to the experimenter, however, some raw data do deserve to be discarded because of unavoidable experimental glitches, but preconceived notions shouldn't drive the data.)

In Part I of this essay, I noted that there is no greater scientific misconception among the general public than what constitutes experimental uncertainty. Scientists are partly to blame, because uncertainties are called errors in research parlance. Tell someone that your experiment had errors, and nobody will believe your result. Tell someone that your experiment had quantifiable uncertainties, and the scientific enterprise is salvaged.

A few years ago, new and improved data led to the announcement that the oldest stars in the galaxy had been born about 14 billion years ago and that the age of the universe was about 12 billion years. The press, invoking the you-can't-be-older-than-your-mother principle, portrayed the news as a cosmic controversy of the first rank.

But if the reported numbers had been accompanied by the scientists' published uncertainties--known as "error bars"--a sensible picture quite undeserving of headlines would have emerged. The age of the universe had been calculated at 12 [ or -] billion years, the age of the oldest stars at 14 [ or -] 2 billion years. The error bars suitably overlapped at 13 billion years, the point toward which current estimates of the age of the universe and the age of the oldest stars are converging.

For decades the mass media ignored the significance of measurement errors. But nowadays, reports of public opinion polls are accompanied by "margins of error"--the sociologist's error bar. If opinion pollsters query only one hundred of the 270 million or so people m the United States, they will (or had better) frame their claims with fat uncertainties. So when a news station reports, "The incumbent leads the challenger 54 percent to 46 percent with a [ or -] 5 percent margin of error," then you have my permission to ignore all subsequent discussion of tine significance of the incumbent's lead. Fortunately, there are well-tested statistical methods that account for the size of your polled sample in relation to the size of the total population. Clearly, if all 270 million people were polled, there would be no uncertainties--except for the unavoidable fact that some people change their minds with the breeze.

Often, the most heated scientific controversies take place amid the noise and confusion of messy data. Perhaps the most famous astronomical controversy of the second half of the twentieth century centered on the numerical value of the famous Hubble constant, a measure of the expansion rate of the universe. Poor data allowed two warring factions to arise on opposite sides of the error bars. One group supported H = 100 [ or -] 10. Another group supported H= 50 [ or -] 5.