Development of Engineering Education as a Rigorous Discipline: A Study of the Publication Patterns of Four Coalitions

Journal of Engineering Education, Jan 2007 by Borrego, Maura

D. Type of Contribution

This dimension attempts to capture the approach that coalition authors selected to encourage faculty to use their methodologies, ideas, and results. Although it clearly provides insight into dissemination strategies, the shift over time also shows increasingly rigorous approaches in both the work and the published description of it. Table 5 shows the categories developed to understand this dimension. The overwhelming majority (74 percent) describe the experience of the authors. The title "experience" was chosen from the abstracts themselves:

"This paper discusses first-year experiences in the implementation of . . . at . . ." [42].

"This paper will present the story of the adoption of the. . . program so that others can learn from our experiences. It wall focus on the process that led to rapid adoption of the new curriculum and will point out important steps and pitfalls" [43].

"This paper discusses some experiences in using the Internet and WWW for teaching computing courses" [44].

Table 5 summarizes the results. The category "Experience" also includes publications that review multiple experiences (9 percent) or describe "new paradigms" or un-tested theories (4 percent). These were included in the experience category because it is the least concrete in terms of product or assessment data presented within the publications.

Other authors developed objects or procedures designed to be easily implemented by others, totaling 20 percent of all publications. Instructional software or Web sites (restricted to programs and sites created by the authors) were described by 7 percent of the publications; 5 percent described instructional modules or student activities, which may include software or Web sites in addition to other resources. Another 5 percent described a matrix or procedure that could be easily adapted to other institutions, but generally did not require any objects, just the knowledge included in the publication itself. Finally, 3 percent described survey instruments or assessment tools, including several concept inventories.

Only 4 percent of the publications mentioned theory from the literature, described experiments designed with control groups, or reported on statistical data analysis. Only one of these three conditions needed to be met for the publication to be classified as research. Scientific Research in Education [4] identified six criteria for rigorous education research, all of which should be met by a rigorous study (so the current analysis may overestimate the number of research publications). Among the criteria are: linking research to theory, using appropriate empirical methods, disclosing research, and facilitating replication across studies. Requiring controlled experiments and quantitative results are not requirements of the authors, nor are they in the opinion of this author. They were simply the only evidence provided in the database abstracts that linked to existing theory or disclosed methods and results for replication. Many of the "experience" publications also included assessment data, but the principal focus was on showing a positive response to the innovative program rather than objectively comparing different groups or treatments [6, 45]. Many of these experience studies could be made more rigorous with a thorough examination of the logic and potential confounds of the intervention, to explain the results in the context of existing theories. The key is to build on prior work, replicating and generalizing to refine theories-not to abandon all past efforts, as the rhetoric of the rigorous research movement occasionally implies.