Trends in research design and data analytic strategies in organizational research

Journal of Management, Spring, 1995 by Eugene F. Stone-Romero, Amy E. Weaver, Jennifer L. Glenar

PUI values for studies using other designs (e.g., literature reviews, meta analyses) remained fairly low for the entire 1975-1993 period, the average PUI being 13.37% for this period. Note, moreover, that the PUIs were below 24% for the entire period, reaching a peak of 23.08% in 1983. Interestingly, as the [TABULAR DATA FOR TABLE 2 OMITTED] results in Figure I show, there appeared to be an increase in the use of other designs over the study period. Consistent with this apparent trend, the correlation between time and the PUI for other designs was .71 (p [less than] .01).

Data Analytic Strategies Used in Coded Articles

Rows 5-21 of Table 1 show PUIs for data analytic strategies used in the 1975-1993 period. Figure 2 plots the yearly PUI levels for articles that tested for univariate or multivariate mean differences. The PUIs for MANCOVA, ANCOVA, and MANOVA were generally below 10% for the entire period. Overall, however, there was a slight, but fairly steady increase in the use of the MANOVA strategy over the 1975-1993 period: The correlation between time and the PUIs for MANOVA was .84 (p [less than] .01). There was also an increase in the use of the ANCOVA strategy over the study period: The correlation between time and the PUIs for ANCOVA was .74 (p [less than] .01).

The mean PUIs for the MANOVA, ANCOVA, and MANCOVA strategies across the 1975-1993 period were 7.93%, 2.85%, and 0.73%, respectively (see Table 1). Compared to these PUIs, the mean PUIs for the t test strategy and the ANOVA strategy were considerably greater, i.e., 21.67% and 35.51%, respectively. Note, however, that over time there was a general decrease in the use of the ANOVA strategy: As the results in Table 2 indicate, there was a -.54 (p [less than]; .05) correlation between time and the PUIs for ANOVA. Note also that although the mean PUI for ANOVA was 38.42% for the 1975-1984 period it was only 32.05% for the 1985-1993 period, a statistically significant decrease (Z = 2.91,p = .002). In addition, the 1993 PUI of 35.71% for ANOVA was considerably below the 1980 high of 46.15% for this strategy.

Figure 3 displays PUIs for selected correlation-based data analytic strategies (i.e., zero-order correlation, multiple regression, classical path analysis, and factor analysis). As can be seen in this figure, the PUI for zero-order correlation varied considerably about a mean of 42.09% for the 1975-1993 period and equaled 38.78% in 1993, about the same as the 1975 PUI of 42.67%. Overall, as the results in Figure 3 show, the use of multiple regression increased considerably over the period considered by the present study: Consistent with this seeming trend, the zero-order correlation between time and the PUIs for multiple regression was .68 (p [less than] .01). The increase in the use of this data analytic strategy is further illustrated by the shift in the 1975 and 1993 PUIs, i.e., 10.67% and 24.49%, respectively. Note finally that although there was quite a gap between the PUIs for zero-order correlation and multiple regression in 1975, by 1993 this gap had narrowed substantially.