Trends in research design and data analytic strategies in organizational research

Journal of Management, Spring, 1995 by Eugene F. Stone-Romero, Amy E. Weaver, Jennifer L. Glenar

Some Other Noteworthy Findings

Two other general trends are worthy of note. First, as can be seen in Table 2, there was a -.64 (p [less than] .01) correlation between the PUIs for ANOVA and multiple regression. As the use of ANOVA decreased the use of multiple regression increased. In view of the fact that neither of these PUIs correlated with the PUIs for either experimental or nonexperimental designs, the -.64 correlation appears to reflect a shift in data analytic strategies rather than a shift in design type. Second, over time there was a trend toward the increased use of multivariate data analytic strategies. This is illustrated, for example, by several correlation coefficients in Table 2; including: (1) the .66 (p [less than] .01) correlation between the PUIs for MANOVA and multiple regression; (2) the .58 (p [less than] .01) correlation between the PUIs for ANCOVA and MANCOVA; and (3) the .72 (p [less than] .01) correlation between the PUIs for MANCOVA and multiple regression.

Discussion

The present study had two major purposes. One was to assess trends in the use of non-experimental versus experimental designs in research reported during the 1975-1993 period. The other was to measure trends in the use of various data analytic strategies over the same period. In order to accomplish these purposes, we coded information from 1,929 articles that were published in the Journal of Applied Psychology during the 1975-1993 period. Results of various analyses showed that although there were differences from one year to the next in the PUIs for nonexperimental and experimental designs, there did not appear to be consistent upward or downward trends in these indices. However, there was an increase in the use of other types of designs over the 1975-1993 period. In addition, the results showed notable changes in the PUIs for several of the statistical procedures that test for univariate or multivariate mean differences, relative stability in the PUIs for zero-order correlation, an increase in the PUIs for multiple regression and several other multivariate statistical procedures, and substantial growth in the PUIs for CSA-based data analytic techniques (e.g., EQS, LISREL). Implications of these findings are now considered.

Declines in the Use of Speciffc Data Analytic Strategies

Simple data analytic strategies. The PUIs in Table 1 and the correlation coefficients in Table 2 showed a gradual decline in the use of some of the simpler data analytic procedures as the major means of testing hypotheses or answering research questions. For example, over time the use of ANOVA decreased somewhat. Although the PUIs for ANOVA have shown a general decline over the 1975-1993 period, there was an increase in the PUIs for MANOVA over the same period. More specifically the PUIs for MANOVA were 4.00% in 1975 and 12.24% in 1993. There are several possible causes of this increase. One is that researchers are becoming increasingly sensitive to the need to consider: (1) the effects of treatments on multiple dependent variables; and (2) the correlations between such variables when assessing the effects of experimental treatments. Unfortunately, we have no direct evidence to support this speculation about the cause of the increase in the use of MANOVA. However, the explanation seems reasonable in view of the fact that the correlation coefficients in Table 2 show that the PUIs for several multivariate procedures increased concomitantly over the 19 year study period.