User-based data collection techniques for evaluating networked information services - Libraries and the Internet: Education, Practice & Policy

There are a number of advantages to using this technique. First, and perhaps most importantly, the participants write, in their own words, their views on the topic being discussed so that the moderator does not have to do so from his or her notes at a later time. Second, this approach allows participants to be informed by the discussion and to modify their views in light of how the discussion evolves. Finally, writing one's views on the handout usually results in a 100 percent response rate from participants--which rarely occurs in surveys or in group discussions where a small number of participants can dominate the conversations.

Site Visits

Site visits are similar to a case study approach (Yin, 1989) except that site visits are not likely to be as time consuming and detailed. Generally, case studies have some longitudinal dimension to them since they are conducted over a period of time. A site visit, however, entails less time and is a bit more informal. A site visit generally is planned to obtain first-hand information from tours of specific facilities and services, interviews with individuals or groups, or observation of specific activities at the site. In addition, another aspect of the site visit is also to obtain reports, brochures, and examples of products or services made available at the site. An interesting aspect of site visits is the potential to directly compare and contrast different types of data collection techniques from different sources on the same topic.

In site visits, it is not always possible to predict in advance the range of data collection activities in which the researcher might engage. Clearly, some of the data collection strategies can and should be planned in advance of the site visit--i.e., scheduling interview times, tours, and so on. Additional data collection opportunities, however, may arise as the site visit progresses. Indeed, the evaluator should be extremely conscious of opportunities to meet with individuals or groups that perhaps he or she could not have known about until the site visit occurred. Two major benefits of such site visits are the opportunity of: (1) having first-hand information about users or activities in a particular setting, and (2) evolving the data collection strategies on site depending on the topics the evaluator deems important to probe for obtaining additional information.

Scenario Development

An interesting but underutilized data collection technique is scenario development. This technique can be done either as a group or as an individual process. The basic idea with this approach is to have participants discuss "what if..." types of questions and construct a scenario or likely series of events that would need to occur if a particular vision or goal is to be accomplished. Scenario development is an especially useful technique for having participants consider possible future events, speculating about what key assumptions may drive the development of future events, and suggesting what designers of networked services and resources might need to consider if they are to be successful in a particular future scenario.