Taking Stock: An Analysis of the Publishing Record as Represented by the Journal of Engineering Education

Journal of Engineering Education, Jan 2004 by Whitin, Katherine, Sheppard, Sheri

(essential in every scholarly paper)

* is well-written;

* has a well-defined goal or thesis and executed in manner consistent with other forms of scholarly research;

* presents a topical and timely subject matter whose impact is felt beyond the immediate future;

* includes convincing data and evidence rather than anecdotal accounts (more than single experience if possible);

(characteristic dependent upon topic)

* identifies and employ specific methods, explaining theory behind analysis techniques where appropriate (e.g., surveys, instruments from other universities or disciplines);

* presents findings based upon a depth of analysis that allows for applicability beyond the specific discipline/scope of the study;

* includes experience as part of analysis (e.g., what worked and what didn't, how methods and results change over the course of the study);

* includes historical perspectives, where appropriate, and demonstrates knowledge of current state of the art in the discipline (e.g., what other schools are doing or have tried, NSF and other programs, technology); and

* involves multidisciplinary authors (e.g., from other engineering disciplines as well as non-engineering areas, such as education, psychology and teaching centers)

The observed characteristics of outstanding scholarly efforts did not drastically change from 1996 to 2001. While the focus in topics of outstanding scholarly papers has shifted over the years, the characteristics that define them have remained constant. An example of a well-executed, scholarly work from 1996 is still such an example in 2002.

The differing characteristics observed were between categories. The variations between categories were not differences in level of scholarship. Instead, there were nuances reflecting the type of topics within each category, which focused research methods and activities in one direction or another. For example, the Assessment and Evaluation scholarly papers were assessing the types of courses and programs discussed in other papers, thereby building on the work of others. The majority of these papers included convincing and comprehensive data and detail and depth in their statistical analyses. References to and explanations of the educational theory behind the study were commonly observed within this category. Finally, a greater number of Assessment papers, relative to other categories were completely authored by individuals outside of engineering.

In contrast, outstanding scholarly papers identified in the Courses and Programs category covered a wide range of topic, resulting in the broadest range of scholarly characteristics. Faculty papers identified were very similar to those coded in Courses and Programs. These papers tended to include more anecdotal accounts of experience within the classroom as well as data. Innovative techniques, used in the classroom for shorter durations with less assessment data, were often reported in Faculty papers.

An important finding from the analysis of scholarly papers is the increase in the number of papers exhibiting these characteristics. The data show a definite increase in percentage; however, given that assessment of "scholarship" is subjective, the numbers are not reported in this study. Those papers published earlier in the study years tended to exhibit a few of the characteristics. As the years progressed, more papers included a larger complement of the characteristics. This was especially noteworthy in the Faculty and Courses and Programs papers that increasingly reported valid assessment data.