Trends in research design and data analytic strategies in organizational research

Journal of Management, Spring, 1995 by Eugene F. Stone-Romero, Amy E. Weaver, Jennifer L. Glenar

Method

Sample

Data used in the present study were derived from content analyses of 1,929 articles published in the Journal of Applied Psychology during the 1975-1993 period (total of 19 years). The number of studies published per year ranged from 78 to 150, and averaged 101.53. Table 1 shows sample sizes for each of the years in the period.

Measures

Each article was coded in terms of several criteria: The first criterion was basic design. The categories under this rubric were: (1) experimental studies, including randomized or true experiments (including statistical simulations) and quasi-experiments; (2) nonexperiments or passive observational studies; and (3) other designs, including meta-analyses, narrative literature reviews, and comments. Note that a single article might report the results of research using more than one design. For example, a multi-study article might consider the results of both a true experiment conducted in a laboratory context and a nonexperiment conducted in a field setting. In cases of multi-study articles we coded each design type that was used.

[TABULAR DATA FOR TABLE 1 OMITTED]

The second basis for categorization was the principal data analytic procedure(s) used in the study. Here the coding was restricted to procedures that were directly relevant to testing a study's hypotheses or answering its research questions. Thus, for example, in a study that used multiple regression to test hypotheses about relationships between predictor variables and a dependent variable, a researcher might report the means, standard deviations, and reliabilities of measured variables. However, for purposes of the present study, only the regression analysis was coded. All other analyses were regarded as ancillary to the study's main objective(s).

The categorization scheme for data analytic strategies considered the following 15 factors: (1) CSA procedures (e.g:, LISREL, EQS); (2) classical path analysis; (3) zero-order correlation; (4) multiple regression/correlation; (5) canonical correlation; (6) discriminant function analysis and multiple discriminant function analysis; (7) factor analysis (common factor and principal components); (8) cluster analysis; (9) analysis of variance (ANOVA); (10) analysis of covariance (ANCOVA); (11) chi-square based tests of association; (12) multivariate analysis of variance (MANOVA); (13) multivariate analysis of covariance (MANCOVA); (14) t tests of mean differences; and (15) other data analytic strategies (e.g., multidimensional scaling, nonparametric analysis of variance). Of course, in many instances more than one data analytic strategy was used in a given study. In such instances coding was performed for all principal strategies used.

Annual percentage use indices were computed for each of the design types and data analytic strategies. In these indices the frequency of use of a given type of research design or data analytic strategy was divided by the number of articles published per year and the quotient was multiplied by 100. For example, the percentage use index (PUI) for nonexperimental designs for 1975 was (88 / 150) x 100 = 58.67%. Note that a PUI can range from 0 to 100 and it reflects the percentage of articles that used a particular type of design or data analytic strategy in a given year.