Assessing the Math Online tool: A progress report

Mathematics and Computer Education, Fall 2003 by McSweeney, Laura, Weiss, Joan

4. The Math Online Study

A comparative study was designed to test the effectiveness of the Math Online system. Although the study has a number of faults we found the results encouraging and worth further investigation.

During most academic years our Applied Calculus I & II, MA 121 (fall) and MA 122 (spring), sequence is offered in nine to fifteen sections with typically between twenty-five to thirty-five students, each taught independently by a separate instructor. Most students complete the sequence with the same instructor during one academic year. Each instructor involved in the study taught two sections of Applied Calculus, one was randomly assigned to utilize Math Online and the other was designated as a control section, in which the instructor taught and assessed in his traditional manner. Seven instructors participated in the study in Fall 2000 with six of these continuing in Spring 2001. In Fall 2001 five instructors participated with four repeating in Spring 2002. Instructors received about two hours of training in the use of Math Online each year. The students were not informed ahead of time in which type of section they had enrolled. This randomized comparative design helps discern any statistical differences achieved from the use of Math Online by reducing the variability introduced by individual instructors. In all sections each instructor followed the same syllabus, but in the Math Online section most of the practice and quiz assessments of a student's algebraic skills and the basic computational skills and formulas in calculus were done outside of class using Math Online.

There was an unexpected external constraint. Traditionally this Applied Calculus sequence had been taught by full-time faculty, but during the last several years due to increased enrollments, most sections of these courses are now taught by adjunct (part-time) faculty. Only one participant each year was a full-time faculty member. Thus most instructors participating in this experiment were adjunct faculty who were significantly less technologically sophisticated and pedagogically experimental than our typical full-time faculty member.

This type of experimental design is not typical. In studies that test new pedagogy, the first users are usually the developers and thus more knowledgeable and enthusiastic about the new technique causing excellent initial results. In many instances, when others get involved the positive effects disappear. In this study only one of the ten different participants was involved with the design of Math Online. Even under these less than ideal circumstances the results of this study indicate Math Online shows the potential to improve students' basic algebraic skills via the web outside of class time.

5. Description of the Data Collected to Assess Math Online

Two types of data sets were collected to assess the students' learning of basic algebraic and computational calculus skills. One set consisted of scores on a pre-test and post-test of fifteen multiple-choice questions on algebra and pre-calculus content. The second set was scores on common calculus exam questions which were included on all semester and final exams and were graded according to a predetermined rubric.