Response rates and response bias for 50 surveys of pediatricians

Health Services Research, Feb, 2005 by William L. Cull, Karen G. O'Connor, Sanford Sharp, Suk-fong S. Tang

A survey's response rate is a conventional proxy for the amount of response bias contained in that study. While there are more theoretical opportunities for bias when response rates are low rather than high, there is no necessary relationship between response rates and bias (O'Neill, Marsden, and Silman 1995; Asch, Jedrziewski, and Christakis 1997). A review of physician surveys published between 1985 and 1995 found an average response rate of 61 percent for all surveys of physicians and an average response rate of 52 percent for surveys with more than 1,000 observations (Cummings, Savitz, and Konrad 2001). There was no significant decline in response rates to published physician surveys across that time, but it is unclear whether that pattern has continued.

By definition, not much information is available about nonrespondents. It has been estimated that only 18 percent of physician survey articles perform any type of comparison between responders and nonresponders with the remainder having only response rates to serve as an indicator of bias (Cummings, Savitz, and Konrad 2001). Direct examinations of response bias for health care professional groups have generally found only minimal amounts of response bias in surveys (Barton et al. 1980; Hovland, Romberg, and Moreland 1980; Locker and Grushka 1988; McCarthy, Koval, and MacDonald 1997; Thomsen 2000; Kellerman and Herold 2001). Surveys of patients and the full population have consistently shown substantially more response bias (Sheikh and Mattingly 1981; Benfante et al. 1989; Brennan and Hoek 1992; Diehr et al. 1992; Vestbo and Rasmussen 1992; Prendergast, Beal, and Williams 1993; Walsh 1994, Blanker et al. 2000; Kotaniemi et al. 2001; Barchielli and Balzi 2002; Fowler et al. 2002; Mazor et al. 2002; Solberg et al. 2002; Partin et al. 2003; Van Loon et al. 2003). Substantial response bias greatly limits the generalizability of survey findings (Asch, Jedrziewski, and Christakis 1997; Cummings, Savitz, and Konrad 2001). Studies focused specifically on assessing response bias for physician surveys are lacking. It is important to determine if response rates for physicians are falling, and if decreasing response rates are associated with increasing response bias.

The American Academy of Pediatrics (AAP) has conducted many surveys of pediatricians through its Periodic Survey of Fellows and other programs. The AAP also maintains an administrative database that contains demographic information about its members. Using this database along with data from the American Board of Pediatrics administrative database, age, gender, membership type, and location data were merged with respondent/ nonrespondent information for 50 surveys of pediatricians. The objectives in doing this were (1) to monitor response rates across time, (2) to examine systematic response bias for several different pediatrician characteristics, and (3) to explore response bias as a function of survey response rates.

METHODS

A total of 63,473 cases were gathered from 50 different surveys of pediatricians conducted by the AAP since 1994. Thirty-one surveys targeted random samples of active U.S. members of the AAP (AAP Periodic Survey and AAP Medicaid/SCHIP Participation Survey), six targeted random samples of pediatric residents (AAP Third Year Resident Survey), and the remaining 13 targeted AAP-member and nonmember pediatric subspecialists (Future of Pediatric Education II Survey).

Between four and six mailings were conducted for each of the 50 surveys and response rates were tracked. A simple definition of response rate was utilized in this study. The number of surveys returned with valid responses was divided by the total number of pediatricians on the mailing list. This provides a conservative estimate of response rate that does not try to adjust the denominator post hoc to exclude doctors whose addresses were bad or whose survey responses indicated that they should not have been included in the target sample.

Amount of response bias was computed for four different characteristics of physicians: age, gender, geographic location, and AAP membership type. These four characteristics were chosen based on their availability in the administrative databases and their usefulness in published analyses. Many articles based on these surveys reported descriptive results for the four characteristics, and the majority of these articles also reported significant relationships between the characteristics and other survey topic questions (Campbell et al. 1996; Brotherton, Tang, and O'Connor 1997; Olson, Christoffel, and O'Connor 1997; Schaffer et al. 1998; Barnett, Duncan, and O'Connor 1999; Brotherton, Mulvey, and O'Connor 1999; Cheng et al. 1999; Schanler, O'Connor, and Lawrence 1999; Redding et al. 2000; Stoddard et al. 2000; Cull et al. 2002, 2003a, b; Pan, Cull, and Brotherton 2002; Pletcher et al. 2002; Wiley et al. 2002; Tunkel et al. 2002; Anderson et al. 2003; Kelly et al. 2003; Kline and O'Connor 2003; Tang, Yudkowsky, and Davis 2003). Of course, there are many other practice characteristics that we did not have access to that also could be associated with survey participation.