Bridging diverse institutions, multiple engineering departments, and industry: A case study in assessment planning

Journal of Engineering Education, Apr 1998 by McMartin, Flora, Van Duzer, Eric, Agogino, Alice

Student portfolios, focus groups, and interviews were also identified as important tools, but were judged too time intensive to be used as broadly as the core tools.* Once the four core assessment tools were selected, the assessment team researched various formats to maximize their validity and limit any disruption to classroom activities.

Scenario based assignments associated with learning outcomes for open-ended problem solving, multi-disciplinarity, and hands-on learning, were designed to be short openended problems based on the scenarios provided by the MIB. These assignments were designed as homework problems to be administered in the first and last week of a semester or quarter. (Several versions of the writing prompts for the scenario assignments are currently being pilot tested to determine the optimal format.)

Design project reports provided the broadest coverage of the abilities, and required the least additional effort by faculty. These reports were based on sample assignments provided by project leaders. (The faculty participated in finalizing a common format for these assignments at the project leader meeting in the summer of 1997, and it is being pilot tested this fall.)

Self-peer reports ask students to assess their own performance and that of their peers. These assessments focus primarily on teamwork skills, but instruments were developed to other abilities, e.g., hands on learning, which could be used alone or (preferably) in combination with design project reports or taped observations.

Taped observations encompass both audio and video taping but focus on two separate student abilities. Video taped observations were designed to capture student oral presentation abilities. This requires a straightforward process of taping and evaluating student presentations of (primarily) final projects. The analysis of teamwork by way of audio tape required the development of a grading rubric to capture the essence of student abilities through the linguistic content of team discussions. The utility of this approach -including the type of analysis and sampling needed to draw reliable conclusions from the tapes -is currently being tested.

To date, the results of the limited pilot testing is inconclusive. Scenario assignments and audio taped team meetings have been tested at two different schools on freshman and sophomores. Industry representatives have completed several of the assignments, and their responses will be used to validate the scoring rubric for the tool and provide baseline data against which to assess student abilities. Final revisions to the tools will be made in the spring of 1998 based on the results of the fall pilot tests for each tool.

The core assessment tools will tell us what happened in terms of student ability development, but not why. To uncover the elements of these courses that are most effective in facilitating student development and how they influence student attitudes, a small sample of students will be recruited on each campus to create student portfolios and participate in focus groups, interviews and attitudinal surveys.