Context effects on examinations: the effects of time, item order and item difficulty

Canadian Psychology, Nov 1998 by Zumbo, Bruno D, Perlini, Arthur H, Lind, David L

Abstract

Four studies were conducted to test the effects of contextual factors -- time, item, chapter and difficulty arrangement -- on test performance. In Experiment 1, we found an advantage for students writing a quiz in the early and middle testing periods for an afternoon group; however, early testing, compared to middle and late testing, was a disadvantage for an evening group writing the same quiz. In a second study, we found that neither chapter retrieval-cues nor sequenced chapter content-order affected exam performance: or that is, exam items forward sequenced with text coverage yielded no advantage over those items that were reverse or random sequenced. In a third study, we manipulated both chapter order (as above) and within-chapter item order so that exam items in any chapter were not sequenced in the same order as text coverage. There was no advantage to receiving exam questions in one sequence over another. In a final study, item arrangement was manipulated according to difficulty level (as described in the text test-bank) so that students were presented with the same set of questions in one of three arrangements: (a) easy-to-hard (EH), (b) hard-to-easy (HE), and (c) random order (R) . The results here indicate no effect for order of item-difficulty on performance. Taken together, the findings indicate that contextual factors like time have an effect on overall performance; however, other factors such as item, chapter and difficulty arrangements have little or no effect on overall performance.

Resume

Quatre etudes ont ete menees pour verifier les effets des facteurs contextuels -- moment, ordre des sujets, ordre des chapitres et organisation des niveaux de difficulte -- sur le rendement obtenu a une epreuve. Dans l'experience n[Symbol Not Transcribed] 1, nous avons decouvert, pour deux groupes soumis au meme questionnaire, que les eleves du groupe de l'apres-midi obtenaient de meilleurs resultats au debut et au milieu de l'apres-midi, tandis que ceux du groupe du soir obtenaient de meilleurs resultats au debut plutot qu'au milieu et qu'a la fin de la soiree. Une deuxieme etude a revele que ni des indices sur les chapitres ni un ordre suivant celui des chapitres n'avaient d'incidence sur les resultats. En d'autres termes, les resultats n'etaient pas meilleurs dans le cas d'un questionnaire suivant l'ordre du manuel par rapport a un questionnaire suivant un ordre inverse ou dont les sujets d'examen etaient presentes au hasard. Dans une troisieme etude, nous avons modifie l'ordre pour qu'il soit different de celui des chapitres (comme ci-dessus) et de celui des sujets a l'interieur des chapitres pour que les questions d'examen ne suivent pas l'ordre presente dans les manuels. Aucune sequence particuliere de questions ne presentait d'avantage quelconque. Dans une derniere etude, nous avons organise les questions suivant leur niveau de difficulte (tel que decrit dans la reserve d'epreuves) pour que les eleves repondent a des questions identiques, mais presentees de trois facons differentes: a) de faciles a difficiles; b) de difficiles a faciles; c) au hasard. Les resultats indiquent que le fait de presenter les questions selon leur niveau de difficulte n'a pas d'effet sur le rendement general. Ainsi, les resultats de ces etudes revelent que les facteurs contextuels comme le moment ont un effet sur le rendement global, mais que d'autres, comme l'ordre des questions, l'ordre des chapitres et l'organisation des niveaux de difficulte ont peu ou point d'effet sur le rendement global.

Based on the conventional wisdom of students and instructors, three common assumptions about classroom assessment were investigated in the present research:

1. Does the time at which a test is completed relate significantly to performance: or that is, does taking a test at the beginning of class lead to better performance than taking a test in the middle or end of class?

2. Do exams in which chapter and/or item content is/are sequenced according to course or text coverage result in better performance than exams in which chapter and/or item content is/are not sequenced?

3. Does the order of item-difficulty (specifically, easy-to-difficult, difficult-to-easy, and random difficulty) have an effect on exam scores?

Experiment 1

Most psychology instructors are familiar with students' interest and enthusiasm over receiving a test or quiz at the beginning, in contrast to later, in the class. This enthusiasm is likely linked to students' concern that the material to be remembered will be forgotten if the quiz in written later in the class, after new material is presented. This assumption appears to have some foundation in the classic memory finding that (a) recall decays over time, and (b) recent information interferes with the recall of previously learned information (i.e., retroactive interference). The more lecture material presented, presumably, the greater the interference.