User-based data collection techniques for evaluating networked information services - Libraries and the Internet: Education, Practice & Policy

Determine Costs and Schedule

For user-based evaluations to be successful, they must be done in a timely fashion and with a clear sense of the costs needed to complete the study. In preparation for the evaluation, costs associated with standard budget items--e.g., personnel, supplies, travel, equipment, contract services--should be identified. There is no use in initiating a user-based evaluation for which there are inadequate resources available to complete it. Indeed, it is better to complete a smaller less costly evaluation than to do none at all.

Equally important is to develop a schedule for the completion of the project and detail the key tasks that will have to be done over the duration of the study. There are many types of GANTT tasking charts and project management software programs currently available that can assist the evaluator in scheduling the evaluation. Such scheduling ensures that everyone involved in the project knows what tasks are to be completed when. Further, scheduling allows the evaluator to monitor the progress of the evaluation more effectively and identify possible problems while they can still be resolved.

Identify the Correct Study Participants

One problem of ten encountered by new evaluators is attempting to obtain assessments from sample participants that may not have the necessary information. For example, in the evaluation of a particular government bulletin board, simply collecting data from a random sample of network users may not produce enough participants that have actually used or know about the bulletin board. The general rule of thumb is to not expect users to provide you with information about things that they know nothing about.

This concern is especially important in focus group sessions. If some members of the focus group are extremely knowledgeable about a particular networked service and others are not, the group dialogue can be extremely skewed. One strategy is to carefully consider whether you need information from naive, beginning, or expert network users. Another is to use a filter question in interviews and surveys to determine the type of user and his or her background before you proceed with collecting the information you require.

Develop, Pretest, and Refine Data Collection Instruments

No data collection instrument should be administered without it first being carefully developed, pretested, refined, and oftentimes pretested a second time. User-based data collection techniques require data collection instruments that make sense to that particular group of users. One useful approach is to have the data collection instruments reviewed by: (1) someone with experience in the data collection technique you wish to employ, and (2) a group of individuals who are members of the user group from whom you will be obtaining data. To ensure reliable and valid data, pretesting and refining of data collection instruments is essential (Kirk & Miller, 1986).

Administer Instruments Successfully