Measuring Change in Sexual Behavior: Concordance Between Survey Measures

Journal of Sex Research, Feb, 1999 by Valerie E. Stone, Joseph A. Catania, Diane Binson

The primary objective of AIDS/STD prevention programs and adolescent pregnancy prevention programs is to change people's sexual behavior. To evaluate whether such programs are successful, a reliable method of measuring how and/or if sexual behavior changes over time is essential. In this paper we investigate agreement between different methods for measuring change in sexual behavior using survey research. Several researchers have looked at the concordance between different survey measures of sexual behavior per se (Coates et al., 1988; Hornsby & Wilcox, 1989; Rotheram-Borus et al., 1994; for reviews, see Catania et al., 1995; Hospers & Kok, 1995); but few have looked at concordance between different measures of behavior change. Measuring change is the methodological question addressed here. To clarify this distinction, a question measuring sexual behavior would ask, "How often do you use condoms?" whereas a question measuring sexual behavior change would ask "Have you changed how often you use condoms?"

In evaluating AIDS and STD prevention programs, different methods have been used to measure sexual behavior change. The most commonly used method, the retrospective change method, asks respondents a question such as "Have you made any changes in your sexual behavior in the past [period of time], and if so, what?" This method is typically employed in studies with a cross-sectional design. Another method, the longitudinal or panel method, asks respondents at two different times about their sexual behavior and compares the two sets of answers. This method is generally used in studies with a prospective design. Studies that use both methods are rare, but these studies provide an opportunity to test the reliability of the two methods of measurement. Comparing respondents' answers on both types of measures can provide concordance data.

Retrospective change measures have been widely used in program assessment. In a review of AIDS prevention studies, Becket and Joseph (1988) cite seven studies using such measures. Several studies designed to assess the effectiveness of various interventions have been published since their review (Baker, Morrison, Gillmore, & Schock, 1995; Baldwin, Whiteley, & Baldwin, 1990; Cochran, Keidan, & Kalechstein, 1990; Keeter & Bradford, 1988; Hingson, Strunin, & Craven, 1989). The Chicago Sex Survey (Laumann, Gagnon, Michael, & Michaels, 1994), which received a substantial amount of media attention in 1994, asked respondents the following retrospective question to assess whether AIDS had any impact on their sexual behavior: "Have you made any changes in your sexual behavior because of AIDS? What have you changed?"

Longitudinal research is often expensive and difficult to carry out. Many prevention programs either do not have the resources available to do a longitudinal assessment or did not plan to do evaluation research when the program started. Even in longitudinal studies, survey researchers may realize in later waves that they want to assess change on a variable that was not asked about in earlier waves. The retrospective change method, therefore, is often the only method available for measuring behavior change. Thus, it is important to examine the concordance between the retrospective change method and other methods of measuring behavior change, such as the longitudinal/panel method.

There are inherent difficulties in measuring sexual behavior accurately, because it generally cannot be observed directly and biases can enter into people's self-reports of their own sexual behavior. Some people are embarrassed about sex or simply feel it is too personal to report on a survey, and therefore may underreport their sexual behavior. Others may be motivated to exaggerate their behavior when reporting on it. Prior research on measurement error in assessments of sexual behavior (item refusal, over- and underreporting, and test-retest reliability) has examined, to some extent, how respondents, instruments, interviewers, and mode of data collection influence measurement error/bias (Catania, Binson, Canchola, & Pollack, 1996; Catania, Binson, Van der Straten, & Stone, 1995; Catania, Gibson, Chitwood, & Coates, 1990; Catania, Gibson, Matin, Coates, & Greenblatt, 1990). For instance, item wording, gender of interviewer, respondent control, and interview mode (telephone vs. face-to-face) have all been found to have some level of impact on response bias to key behavioral assessments in sexological surveys (Catania et al., 1995; Catania et al., 1996; Catania, Gibson, Chitwood, & Coates, 1990; Catania, Gibson, Matin, & Coates, 1990). Where researchers have been able to estimate the bias in measures of sexual behavior, the biases are almost always in the direction that the respondents perceive to be socially desirable.

This past research has generally focused on methodological problems in obtaining point estimates of behavior (although prior investigations on reliability issues touch measurement of behavior change). Methodological issues in assessing changes in sexual behavior over time are key to understanding how populations are altering their sexual behavior in response to HIV/STD prevention programs, other types of sexual health initiatives, and societal and technological change (e.g., Viagra). Any method for measuring change in sexual behavior will have some bias. We have no objective data to test various methods against and, therefore, no way to assess absolute validity of change measures. Social desirability biases are likely to enter into measures of change in sexual behavior as well. Both the retrospective method and the longitudinal/panel method are subject to the problem of social desirability biases.