Developing and assessing statewide competencies for engineering design

Journal of Engineering Education, Apr 1998 by Trevisan, Michael S, Davis, Denny C, Crain, Richard W, Calkins, Dale E, Gentili, Kenneth E

VI. GANG ACCEPTANCE OF THE ASSESSMENT SYSTEM

As the TIDEE project moves toward statewide dissemination of the assessment results with the resulting aura of accountability, discussions regarding the adequacy of the competencies and merit of the assessment system itself have increased. The history of evaluation in education and social programs documents considerable tension when data are used for accountability and program personnel are also expected to use the data for program improvement.9 Much of this tension arises when assessments are developed external to the program. Program personnel often view these assessments and data obtained from them as disconnected from their day-to-day practice and therefore, irrelevant for use in program improvement. Program personnel may also fear the consequences of misusing evaluation data in a high stakes accountability environment. Thus, concern among some faculty within the TIDEE project is no surprise.

For these reasons, programs considering the adoption of the process outlined in this paper should be aware of the possibility that faculty may not buy-in to the assessment system and process. The TIDEE project has actively worked to deal with this issue within the design of the assessment process itself and has realized a benefit of strong faculty buy-in. Significant design features include assessment training for faculty, incorporating faculty into the assessment development process, and developing clear, relevant curricula informed by faculty classroom experience. These design features have merit and are strongly encouraged as features of similar assessment systems.

A. Assessment Training

During the development of the assessment system, the assessment task force was adamant about limiting time devoted to the assessment. This became operationalized as three hours.

In addition, the task force did not want to conduct a direct observation of the team design activity because it was thought that the amount of time needed to conduct direct observation was too costly. This decision was made despite agreement that direct observation of team performance would be the most realistic and perhaps valid approach to the assessment of the teamwork component of the activity. Once data were obtained and examined however, some began to question the validity of using indirect measures, whether the sampling of competencies was sufficient, and therefore, whether or not the results were useful. This occurred despite a seemingly well thought out approach to assessment.

In reflection, those faculty members who questioned the usefulness of a particular approach are implicitly calling into question the standards used to build the assessment.

In order to foster acceptance of the assessment system and ensure that sound decisions can be made from the data, it is essential that assessments meet high standards of quality. The assessment training during the workshops, as well as training provided to the assessment task force, dealt specifically with quality standards for assessment. These workshops provided common language and understanding among faculty to effectively gauge the quality of the assessment system. The workshops are in large part responsible for productive dialog about assessment, its role in engineering programs, and support for the TIDEE assessment system. Therefore, programs adopting our assessment approach must keep vigilance over standards in order to foster utilization of the data, particularly given the fact that assessment is a compromise of alternate assessment strategies and sampling of the content domain assessed. If programs are to be successful in the new accreditation environment, faculty must understand this issue and deal with it effectively. Assessment training has proven to be an effective strategy for ensuring assessments developed by faculty are of high-quality, that faculty understand the difference between sound and unsound assessment practices, and that faculty are skilled in articulating the merit and worth of the assessment system and results.