Trends in research design and data analytic strategies in organizational research

Journal of Management, Spring, 1995 by Eugene F. Stone-Romero, Amy E. Weaver, Jennifer L. Glenar

In planning empirical research, choices have to be made about research design (experimental vs. nonexperimental), research setting (e.g., laboratory vs. natural setting), measures (e.g., questionnaires, observations of behavior), and data analysis strategies (e.g., analysis of variance, multiple regression, covariance structure analysis) and a host of other factors (cf. Kerlinger, 1986; Runkel & McGrath, 1972; Stone, 1978). In the coming years these choices may be influenced increasingly by recent growth in the availability and ease of use of covariance structure analysis (CSA)-based computer programs (e.g., EQS, LISREL) for assessing the plausibility of models that: (1) posit causal relationships between latent constructs on the basis of observed covariances between observed variables; or (2) posit that scores on observed variables are a function of underlying constructs. At present, however, little is known about the effects that such CSA programs have had on the choices that researchers make in planning empirical research. Thus, the major purpose of the present study was to assess trends in research design and data analytic strategies over the time period that immediately preceded and followed the introduction of the LISREL software (one of the two major programs for CSA) by Joreskog & Sorbom (1976).

Causal modeling procedures have been used for several decades in the biological, social, and behavioral sciences (cf. Asher, 1976; Blalock, 1964, 1971; Bollen, 1989; James, Mulaik & Brett, 1982). Initial work on one of the earliest forms of causal modeling, what has been referred to as classical path analysis (Joreskog & Sorbom, 1989), was performed by Wright (1921, 1934, 1960; cf. Bollen, 1989; Joreskog & Sorbom, 1989). This early work was followed by refinements in classical path analysis procedures and the development of a variety of other correlation-based techniques (e.g., partial correlation, multiple correlation, cross-lagged panel correlation) for assessing the degree to which relationships between variables are consistent with an assumed causal model that links the variables (cf. Blalock, 1961; Duncan, 1966; Lazarsfeld, 1948, 1972; James et al., 1982; Kenny, 1979; Rozelle & Campbell, 1969; Simon, 1954, 1971). It deserves noting that subsequent to their development and popularization, several of these techniques have been shown to be ineffective in modeling presumed causal connections between variables and prone to yielding misleading results. For example, Rogosa (1980) demonstrated numerous problems with the cross-lagged panel correlation strategy that was once thought to be useful (e.g., Kenny, 1975; Rozelle & Campbell, 1969) in modeling causal processes using data from longitudinal studies.

In recent years, notable advancements in causal modeling procedures have stemmed from the work of a number of individuals, including Joreskog and his colleagues (e.g., Joreskog, 1970, 1973, 1978; Joreskog & Sorbom, 1976, 1989) and Bentler and his coworkers (e.g., Bentler, 1985). Joreskog and his associates developed structural equation model (SEM)-based procedures and related computer programs that rely on the analysis of observed covariances between measured variables. These covariance structure analysis (CSA)-based programs include the now popular LISREL8 program and its predecessors (e.g., Joreskog & Sorbom, 1976, 1989). Other programs for performing CSA-based analyses (i.e., EQS and EQS/PC) were developed by Bentler (1985).

Analyses performed by the LISREL and EQS programs consider two major issues: One is the extent to which a theory-based model describing hypothesized causal connections between latent variables is consistent with an observed set of covariances between measured variables. Such analyses are concerned with the testing of latent variable models. The second major issue considered by CSA programs is the degree to which observables are a function of a hypothesized set of latent variables. Such analyses are concerned with measurement models. Of course, SEM procedures may simultaneously consider both of these issues. In this case, general or full models are the focus of the analysis (cf. Bollen, 1989; Joreskog & Sorbom, 1989).

The availability of such CSA software as EQS and LISREL may have resulted in changes in the way that organizational researchers (i.e., individuals in such fields as management, industrial and organizational psychology, organizational behavior, organizational theory, organizational communication) and researchers concerned with a host of other phenomena have approached the task of data analysis. More specifically, the existence of CSA-based software may have led to an increase in the use of structural equation modeling (SEM) procedures by researchers.

The existence of such software also may have led to systematic changes in the research designs used by investigators in various academic disciplines. For example, the availability of CSA software may have motivated organizational researchers to use non-experimental designs more frequently than in previous years. Several factors may account for this: First, it is often difficult, if not impossible, to conduct experimental studies in organizational contexts. Thus, nonexperimental studies, often involving questionnaire measures, may be perceived as easier to perform than experimental studies. Second, researchers may be reluctant to perform experimental research in laboratory contexts because of fears that such studies may be viewed as having low levels of external validity and as less likely to be published than the findings of nonexperimental, field-based research. Third, some organizational researchers may assume, quite incorrectly, that it is appropriate to derive causal inferences from studies that use nonexperimental designs if the data from such studies are analyzed with CSA-based procedures (e.g., EQS, LISREL). In view of these factors, the popularity of nonexperimental studies using CSA may have increased in recent years.