Validity of measures is no simple matter

Health Services Research, Oct, 2005 by Lee Sechrest

It is common in published articles in the social and behavioral sciences to encounter a statement to the effect that a measure used in research can be considered "valid." In reviewing research proposals for scientific review committees, statements such as "This measure has been shown to be valid," or "This measure has demonstrated construct validly," abound. The idea of "validity" of measures is often taken to be straightforward, and, indeed, it may be if one confines one's interests to empirical or predictive validity (utility). A measure can be considered to have empirical validity to the extent that it correlates with some other phenomenon in which one is interested. From a conceptual/theoretical perspective this is a nearly trivial case, even though empirical validity may have considerable practical importance.

More often than not, however, when the idea of the validity of a measure is at stake, the interest is in construct validity, a term introduced in 195,5 by Cronbach and Meehl, in one of the most important articles on measurement ever published in the social and behavioral sciences. Construct validity refers to the extent to which a measure reflects accurately the variability among objects as they are arrayed on the underlying (latent) continuum to which the construct refers. Since an underlying or latent variable cannot be directly observed, there is no direct way to determine just how well a measured variable, the one we can observe, maps onto the underlying variable. Thus, there is no way of attaching any numerical quantitative estimate to the idea of construct validity, although we might have a strong sense that construct validity is greater in one case than another. As Cronbach and Meehl (1955) made clear, construct validity of a measure is established by demonstrating its place in a nomological net of consistent, related empirical findings. An impression of construct validity emerges from examination of a variety of empirical results that, together, make a compelling case for the assertion of construct validity for a given measure. Construct validity cannot be assumed simply because a measure correlates with some other measure or because in a factor analysis it seems to have an appropriate factor structure.

Those sorts of things are merely prerequisites, the beginning places for the search for construct validity. The complexity of the construct validity problem is made clear by Messick's (1989, 1995) delineation of six facets of construct validity: content, substantive, structural, generalizability, external, and consequential. (1) Construct validity is not simply a property of a measure but is a reflection of and resides in the conditions of its use.

The issues involved in validity are far too complex to fit well into any simple scheme, and they are remarkably difficult to translate into practice. Even standards such as the Joint Standards (APA, AERA, NCME 1999) for tests proposed by a consortium of professional organizations are difficult to apply and do not necessarily make much sense once one gets beyond the realm of commercially marketed instruments. Achievement of, or even understanding of, construct validity cannot be guaranteed by any template of requirements by which to judge the adequacy of measures, whether those measures are extant, in the developmental process, or merely being contemplated. Rather what is needed is a deeper understanding than seems to be prevalent of what is meant by validity and harder thought about how the measures we work with fit our conceptions of what validity is or ought to be.

The crux of the matter lies in Messick's assertion that "Validity is not a property of the test or assessment as such, but rather of the meaning of the test scores" (Messick 1995, p. 741). That is exactly the problem: it is very difficult to know what the test scores mean. It is not measures that are valid, but the scores that they yield and the interpretations we make of them.

It is important here to note the apparent, but perhaps not so real, difference between the position being taken here and that of Borsboom, Mellenbergh, and van Heerden (2004), who insist that the issues surrounding validity of tests can be made quite simple once one realizes that validity simply means that a construct exists and that the construct causes scores on the test. Their position is that mere correlation between two variables, even if they can be ordered with respect to time or conceptual priority, does not mean that one is a measure of the other. I would grant them their position and commend the clarity of their thought. The complexity with which I am concerned here, however, has a great deal to do with the correct specification or identification of the construct in the first place. That problem is at the core of our difficulties with validity and is, I insist, no simple matter.

THE MEASUREMENT OF BLOOD PRESSURE

Take blood pressure measurement as an example of a measurement task considered pretty well solved by the existence of good instruments. I choose this example because it is not in the realm of social science, perhaps making the issues to be dealt with somewhat starker but less effectively arousing than would an example from psychology or another social science. In fact, however, even more than 100 years after the invention of the sphygmomanometer, problems in measuring blood pressure persist, along with uncertainties about the meaning of the values obtained (Parati 2004). The medical literature on blood pressure measurement and meaning grows by dozens of articles each year.