Does sampling work? - surveys

Business Economics, Jan, 1997 by Barbara A. Bailar

All of us form our impressions of banks, restaurants, foreign countries, businesses, and other people on the basis of a sample of dealing with them. Each of these institutions or people may interact with others many times, but our knowledge is based on a limited set of interactions that we know about. Such samples are not scientific, may be based on very special circumstances, and provide very biased data, but we often act on the data provided. We may no longer do business at a certain bank because a loan officer growled after eating a lunch that caused heart burn or we may recommend a cleaner to all our friends because the clerk behind the counter was so friendly. In any case, we all have experience at acting on the information from a sample.

The sampling I discuss in this paper is necessarily more scientific, because the decisions made on the basis of the data are more important, involving millions of dollars, important public policy, or affecting thousands of citizens or consumers. In the late 1930s and early 1940s, a group of statisticians at the U.S. Bureau of the Census, led by Morris Hansen and William Hurwitz, put sampling on a scientific basis. Most of the samples that underlie public policy today, sponsored by the federal government, are based on probability sampling where each unit of the population has a known nonzero probability of being in the sample before the sample is selected. Some samples, but rarely those in the Federal government, are simple random samples, where each unit has an equal probability of selection. This is not usually cost effective and for samples of businesses, ignores the influence of size of business on the sample estimates. For household surveys, it ignores the cost and inefficiency of a sample that could be scattered everywhere.

Probability samples depend on rules for the selection of the sample but provide benefits such as being able to measure the precision of the sample from the sample units themselves. This enables the analysts to be able to say that, even though the survey results for unemployment may show a difference from the last measurement, the results are not statistically significant. In fact, because the formulas for measuring precision are well known, we can specify ahead of time the sample size needed to attain the precision required.

Most of the large government surveys upon which public policy decisions are made are based on probability samples. In the 1940s, sampling theory was developed, depending on a close connection between theory and practice. Now, the monthly labor force survey, the retail trade survey, the crime victimization survey, and numerous longitudinal surveys are based on probability samples.

It is curious that, with the excellent track record that sample surveys have provided, questions about the use of sampling as a technique still arise. I don't mean that questions about specific sample surveys shouldn't arise; I mean that questions about whether sampling is a valid technique shouldn't arise. A recent instance of those questions being raised are with the proposed use of sampling in the 2000 Census. I will return to this example later in the paper.

Why does sampling make sense? One reason is that it costs less than an entire enumeration. Because the sponsors of a survey can give the precision requirements needed, a survey can be designed with the size requirements in mind. The cost of collecting data from a sample will be lower, in general, than collecting from a complete population. Usually, sample results will be available much more quickly, so timeliness is increased. Response burden to the population of interest is reduced. Usually, a sample is much more tightly controlled than a complete enumeration can be because fewer interviewers are needed, more time can be spent following up on nonresponse, and better supervision can be provided. Also, sometimes a better collection technique can be used in a sample that would be prohibitively expensive if used for everyone. Thus, sampling has distinct advantages as far as cost, size, and quality are concerned.

CHARACTERISTICS OF A SAMPLE SURVEY

Samples have many unique characteristics, the first being the definition of the population of interest. Is the population one wants to describe the entire U.S. population, or the noninstitutionalized population, those who are noninstitutionalized and over eighteen years of age, or those who have a certain type of illness, whether they are institutionalized or not? Or is the population one wants to describe that of all retail stores in the United States, or those retail stores in certain categories, or those that have been in business at least a year, or only those retail stores that do over a certain volume of business each year? The answer to this question of "What is the population of interest?" is a very important one, because it determines how encompassing the sample needs to be and it may determine what can be used as a sampling frame from which to select the sample.