Analytic Assessment of Online Portfolios in Undergraduate Technical Communication: A Model, The

Journal of Engineering Education, Oct 2006 by Johnson, Carol Siri

ABSTRACT

This paper describes an innovative model for assessing the technical communication course by analytically scoring online portfolios, open to the internet, for ten separate (analytic) variables and one overall (holistic) score. The model is a statistically verifiable and sustainable method that strengthens the curriculum and fosters consensus within the teaching community. We achieved construct validity by redefining the elements of the course to incorporate communication in the digital age and then by creating new criteria for evaluation related to that construct. We achieved inter-reader reliability by beginning each assessment with a calibrated reading and by adjudicating non-adjacent scores. After using the model successfully for three semesters, we can see increased consistency in teaching among sections and semesters, more communication among instructors, and we are beginning a database with which we can test further change. The theory and method behind this model can be applied to other disciplines as well.

Keywords: technical communication, assessment, portfolios, online

I. INTRODUCTION

Communication skills are increasingly important in engineering in which specific knowledge must be transferred between culturally different and (sometimes) geographically distant groups. Communication is a large part of the engineering profession, and the future success of students depends on whether they can work with knowledge transfer and knowledge generation. The methods by which engineers communicate are also changing: in addition to using written language, students need to learn electronic and visual methods of communication [1]. Because of these changes, we updated our traditional curriculum, and, at the same time, created a model to assess it. Our assessment process has already fostered more interaction among the students, instructors, and administration and has added consistency to the course. It also provides us with a statistical tool that we can use to query individual components of the course as we seek to continually improve it.

Although we assess individual student outcomes in the portfolios, the results are entered into a database that reflects the program as a whole. The purpose of our program assessment is continual improvement. The ability to incorporate change is essential in a rapidly changing technological environment and this database will help us to monitor the changes that we make. The Accreditation Board for Engineering Technology (ABET) pioneered the idea of program assessment using outcomes as input for further change and thereby provided us with a theoretical model for our cyclic assessment. Our model is based on the ABET Criteria for Accrediting Engineering Programs that mandates each program to have published educational objectives, regular assessment, and evidence that the results of the assessment are applied to the further development of the program [2].

However, to make program assessment sustainable, all parties-faculty, instructors, administrators, and students-must have an active role. In order for assessment to be authentically undertaken, teachers must be involved in the planning and design [3]. Involving multiple stakeholders in the process creates more work for everyone, but the results are worthwhile: it creates consensus among faculty and instructors, yields valuable information to administrators, and ultimately benefits students [4]. Thus, program assessment must be site-based and locally controlled in order to be successful [5]. Each program in each discipline could theorize and implement a homegrown assessment thereby rendering third-party accountability testing unnecessary.

In our case, technical communication students create an individual online portfolio that contains their work for the semester. These portfolios are open to the internet and hosted on university servers. Each semester, faculty and instructors gather together to read and assess a random sampling of the portfolios. We score the portfolios analytically for ten separate communication criteria and holistically for one overall portfolio score. Creating and implementing this model took place gradually, since the faculty and instructors had to change their curriculum to include basic web work and, in so doing, often had to learn basic web design and posting skills themselves. This form of online portfolio assessment, using both analytic and holistic scoring with portfolios open to the internet, is new and holds promise for future development in multiple disciplines.

II. BACKGROUND

New Jersey Institute of Technology (NJIT) is a comprehensive technological university that offers 34 undergraduate degrees, 40 master's degrees and 19 doctoral programs. Undergraduate enrollment in 2004 was 5,336, the majority of whom were in engineering or computing sciences. Our university is one of the most diverse in the nation: undergraduate enrollment in 2004 was only 34% Caucasian, and the number of American students who speak English as a second language is approximately half [6]. Thus, we face a dual challenge: we must teach students from a variety of linguistic backgrounds to communicate effectively across cultures, in addition to the usual task of teaching them to communicate their disciplinary knowledge clearly.