Introducing peer review in an IS analysis course

Journal of Information Systems Education, 2003 by Sindre, Guttorm, Moody, Daniel L, Brasethvik, Terje, Solvberg, Arne

ABSTRACT

Peer reviews were introduced as a teaching technique for the 2002 offering of the course SIF8035 Information Systems at the Norwegian University of Science and Technology (NTNU). The students handed in conceptual models which were then double-blind reviewed by three independent peers. The review reports contained scores and defect lists, but were not used for grading. This paper conducts an analysis of the outcomes of the peer review exercises. Several approaches are used, both quantitative and qualitative, investigating the students' performance and perceptions. The main conclusions are: 1) The scores given by the peer reviewers were not reliable enough to recommend their use for grading purposes. 2) The introduction of peer review exercises contributed positively to students' learning in the course - but not equally so for all students. A substantial fraction of the students did so little that it is hard to claim any learning effect. The main reasons for this seem to have been poor motivation and unclear demands. In hindsight, we have discussed the distinguishing properties of three different purposes of peer-reviews that were not clear to us when the course started, and we have identified several possible ways of improving the peer reviews. In spite of the reported problems, experiences are more positive than negative, and it has been decided to continue with peer reviews in the course.

Keywords: information systems analysis, conceptual modelling, peer-review, learning effectiveness

1. INTRODUCTION

In information systems development, it is a highly recommended practice to assure the quality of artefacts on all levels before they are used further on in the project. Such quality assurance activities, commonly known as inspections or reviews (Gilb and Graham 1993; Wiegers 2001), have been found to discover product defects far more effectively than testing of the developed code. The most effective discovery of defects occurs if inspections are applied systematically already at the requirements level (Kelly, Sherif et al. 1992). However, in the education of information systems engineers, review and inspection techniques have largely been neglected (Johansson 1997). This was also the case in the course SIF8035 Information Systems at the Norwegian University of Science and Technology, until the spring term of 2002. Then, a major change was made to the exercise part of the course, introducing peer-reviews of conceptual models. This paper constitutes a post-course analysis of this pedagogical intervention, trying to answer the following questions:

1. Would it have been reasonable to use the scores from peer-review for grading? (The peer scores had no impact on final grades in this offering of the course, but the 2002 offering could partly be considered a trial run to investigate whether peer grading would be a way to go.)

2. Did the introduction of peer reviews improve the students' learning in SIF8035 Information Systems?

3. Would it have been better if peer reviews had been introduced in another way?

4. Should peer reviews be included in IS Analysis and Design courses in general, and what teaching guidelines can be extracted from our experience?

These questions are listed in increasing levels of ambition. The first can be fairly well answered from our data, while the answers to the latter two will mostly be subjective opinions, although based on experiences from the review exercises. The rest of the paper is structured as follows: Section 2 discusses various purposes of peer-review in education. Section 3 presents the course as it was before peer-review was introduced. Section 4 describes the peer-review activities in more detail. Section 5 presents research methods and results. Section 6 discusses the interpretation of the results. Section 7 makes the conclusion of the article, whereupon section 8 discusses possibilities for further work.

2. DIFFERENT PURPOSES OF PEER REVIEWS

In education, peer-review would mean that students evaluate each other's work. (Newell 1998; Eschenbach 2001). Peer- review can be introduced for several different purposes:

* (Peer) Reviews may be a topic in the course. Peer is put in parentheses here, since in industrial IS development projects, reviewers may sometimes have quite different roles and backgrounds than those who wrote the documents submitted to review (e.g., a requirements specification may be written by analysts but reviewed by customer representatives).

* Peer-review may have positive pedagogical effects.

* Peer-review may reduce the work burden of the staff.

Of course, one may have all these three motivations at once. But there are some potential conflicts between them. As argued in section 1, review techniques may be a legitimate topic in an IS course, a learning goal then being to train the students to become good reviewers in later work-life. With this review-as-topic outlook one might expect to see that

* The review exercises are backed up by theory on review techniques, through lectures and reading materials.