Trends in research design and data analytic strategies in organizational research

Journal of Management, Spring, 1995 by Eugene F. Stone-Romero, Amy E. Weaver, Jennifer L. Glenar

Shifts in prevailing designs and data analytic strategies are important for several reasons. First, such shifts may signal the need for changes in the content of university-based training programs. For example, courses in CSA procedures might be required of all individuals in graduate-level training programs in the organizational sciences (e.g., management, industrial and organizational psychology, organizational theory, organizational communication, and organizational behavior) and other disciplines. Second, as a result of such shifts, more emphasis might be placed on nonexperimental research designs and strategies in graduate-level research methods courses in various disciplines, including the organizational sciences. Third, to the degree that CSA procedures are used by researchers, "gate-keepers" of various types (e.g., journal editors, program chairs of conferences of scientific societies, members of editorial boards, reviewers of conference papers) will need to have sufficient familiarity with both CSA methods and the characteristics of nonexperimental research to provide competent evaluations of papers that are submitted for review.

Unfortunately, at present, there are no data on the extent to which CSA procedures are being used by organizational researchers. Moreover, there are no data on the degree to which research designs of various types are being used by organizational researchers. Thus, the overall purpose of the present study was to provide answers to two major questions: First, has the availability of such techniques as path analysis, LISREL and EQS led to changes (over time) in the way that data are analyzed by researchers in the organizational sciences. Second, have the basic designs used by researchers in the organizational sciences shifted over time? In order to provide answers to these questions we conducted a content analysis of articles published in the Journal of Applied Psychology for the 1975-1993 period. The results of this analysis were used for two major purposes. First, we assessed trends in the use of basic research designs (i.e., experiments, nonexperiments, and other designs) over time. Second, we appraised trends in the use of data analytic strategies that involve: (1) the comparison of group means (e.g., t tests of mean differences, analysis of variance, multivariate analysis of variance); (2) the assessment of relationships between two or more variables (e.g., zero-order correlation, multiple regression); and (3) the analysis of covariance structure data (e.g., testing of structural equation models). The major interest in these assessments was to determine if the frequency of use of given procedures and/or designs has increased or decreased systematically over the 1975-1993 period. To assess this we computed zero-order correlation coefficients between the year articles were published (time) and annual usage indices (described below) for various research design and data analytic strategy types. In addition, we examined yearly usage data for high and low points. Moreover, in instances where there were noticeable shifts in usage we tested for the presence of changes using appropriate statistical techniques (e.g., test of differences between proportions). These tests considered either selected pairs of years or selected sets of years for which there were apparent differences in usage.