Conducting Web-Based Surveys

ERIC Educational Reports, Dec 01, 2001 by David J. Solomon

The growth of the Internet has impacted on virtually every aspect of society. Survey research is no exception. Two years ago in an informal search of Yahoo, Kay and Johnson (1999) identified over 2,000 Web-based surveys(1) in 59 areas. The interest in Web-based surveying is not surprising as it offers a number of distinct advantages over more traditional mail and phone techniques. Examples include reducing the time and cost of conducting a survey and avoiding the often error prone and tedious task of data entry (Medin, Roy & Ann, 1999).

Email offers one option for distributing Internet surveys. Up until a few years ago email surveys were the predominate means of Internet surveying. As the World Wide Web (WWW) has grown in popularity, the use of Hypertext Markup Language (HTML) forms or Web-based surveys are becoming the dominant method of gathering survey data. These forms streamline the data collection process formatting and entering responses directly into a database for analysis. Since HTML forms can be made programmable, it is also possible to have real time error checking and correction increasing the accuracy of the data collection process. The formatting capabilities of HTML allow the creation of easy-to-read and a attractive forms that may improve response rates. In addition, the programmability of HTML forms makes it possible to randomly order responses and tailor options based on information the respondent supplies earlier in the survey.

Combining an email "cover letter" as a means of contacting sampled people with the use of an HTML form for data collection provides an especially effective and efficient approach to Internet surveying. Modern email packages automatically convert universal resource locators (URLs) or web-addresses in the text of an email into a hyperlinks. Placing the URL of the survey form in a cover letter email allows the respondent to "click" their mouse on the URL to display the survey form and subsequently fill it out. CONCERNS WITH WEB-BASED SURVEYINGAlthough Web-based surveying is very attractive, at this point it should be used with caution. Currently the biggest concern in Internet surveying is coverage bias or bias due to sampled people not having or choosing not to access the Internet (Kay & Johnson, 1999; Crawford, Couper & Lamias, 2001). Despite expediential growth of the Internet there are still large numbers of people who do not have access and/or choose not to use the Internet. It is also clear that there are wide disparities in Internet access among ethnic and socioeconomic groups (Selwyn & Robson, 1998).

There are specific populations where Internet access is extremely high and coverage bias is likely to be less of a concern. College students and university faculty within the USA, Canada and Western Europe are examples of such populations. Even though coverage bias may be less of an issue in these groups, experience and comfort with Internet-based tools such as Web browsers is another serious potential source of bias both in response rates and the way people respond to the survey (Dillman, Tortora & Bowker, 2001).

Web-based surveying is still in the early stages of development. The WWW is a unique media and it is not clear to what extent the knowledge we have gained over years of experience with more traditional surveying techniques fully applies to Internet surveying (Dilman, Tortora & Bowker, 2001). Studies are just beginning to done to learn the optimal ways to structure and format Internet surveys to limit biases and increase response rates. It is also likely that the best way to design an Internet survey depends in part on the familiarity and comfort of the respondents in using Web browsers and email clients. It is also quite likely that the type of Internet connection as well as the hardware and software used in accessing the Internet will impact on response rates and possibly how a person responds to an Internet-based survey.

The use of HTML forms for surveying poses a unique set of issues and challenges that need to be addressed to ensure valid data. The Web is a very public place and unless steps are taken to limit access to a survey, it may be found and responded to by people who are not among those sampled by the researcher. This can either happen by accident or maliciously. Since one only has to "click" their mouse pointer on the "submit" button to respond to a Web-based survey instrument once it is filled out, it is also quite possible for respondents to either mistakenly or purposefully submit multiple copies of their responses.

While Internet-based surveying techniques need to be used with caution, their benefits warrant continued exploration and the cautious use. It is also pretty clear that coverage bias and familiarity with Internet tools will be less and less of an issue over time. Additionally our knowledge about how best to conduct Internet surveys will continue to improve with research and experience. RESEARCH ON INTERNET-BASED SURVEYINGAlthough the research on Internet-based surveying is limited, findings are beginning to appear in the literature. Several studies have found that response rates for Internet surveys are lower that equivalent mail surveys (Medin, Roy & Ann, 1999; Cooper, Blair & Triplett, 1999). As noted by Crawford and colleagues (2001), this may be due to our lack of knowledge on how to achieve high response rates using the Internet surveys. The lower response rates for internet surveys may also reflect coverage bias, the lack of familiarity with the media and/or lack of convenient access to the Internet. In the author's experience, Web congestion can also be a factor in lowering response rates for Web surveys particularly with people who have relatively little experience with the Internet.Cook and colleagues (2000) conducted a meta-analysis of factors influencing response rates in Internet-based surveys2. They found that follow-up contacts with non-respondents, personalized contacts, and contacting sampled people prior to sending out the survey were the three dominant factors in higher response rates. Kittleson (1997) in a study of email-based surveying found it was possible to double the response rate with follow-up memos though in general this may be somewhat optimistic. As with mailed surveys, repeated follow-ups have diminishing returns and at some point risk irritating potential respondents without noticeably increasing response rates. Additionally, Dillman, Tortora, Conrad & Bowker (2001) found that relatively plain Web surveys that load quickly resulted in higher response rates than "fancier" surveys that take longer to load. Jeavons (1998) analyzed detailed server logs from three separate large-scale surveys. He found a relatively high percentage of potential respondents stopped completing the surveys 1) when encountering the first question, 2) when encountering a complex question grid, and 3) when asked to supply their email address. This suggests that some potential respondents have difficulty with the media and give up early in the process of completing the survey or when encountering complex questions. Others may be reluctant to give out personal information such as an email address. The logs were also merged with demographic data collected via the surveys. Somewhat surprisingly no patterns in failure to complete rates were found by gender, age or education level. In two of the surveys, people with lower income were found to have a higher rate of repeating screens of questions mainly due to improperly filling out questions. DEVELOPING WEB SURVEYSAs noted, due to their inherent advantages, most Internet surveying is now being done using HTML forms with potential respondents often contacted via email cover letters. While some developers still directly code these forms in HTML, there are dozens of HTML editors available, and they are becoming increasingly sophisticated and easy to use. There are two general methods of capturing the data entered into an HTML form. The form can be programmed to email the data back to a specified email address or captured by a program on the server called a common gateway interface (CGI) script. Using CGI scripts is more robust, offers more flexibility and is the far more commonly used method of capturing data. There are several HTML development packages that both provide HTML editing capabilities and automate the process of developing the CGI scripts necessary to capture data from HTML forms developed with the package. Two widely used examples of these packages are Microsoft's FrontPage and Macromedia's ColdFusion. While these packages are general purpose Web development tools, there are also a growing number of software development systems designed specifically for Web-based surveying. Examples include Perseus's Survey Solutions for the Web, Creative Research System's The Survey System, and Survey Said Survey Software. These packages tend to offer additional features specific to survey research. Examples include as managing the distribution of email cover letters, built-in statistical analysis and reporting capabilities, and automatic tracking of people who have responded coupled with the ability of sending out follow-up email reminders to those who have yet to respond. Their HTML editors are also geared for survey form development, allowing them to simplify and streamline the process of developing and formatting the question response fields. Web Survey Mailer System