Assessing Students' Metacognitive Skills

American Journal of Pharmaceutical Education, 2007 by Garrett, Judy, Alman, Martha, Gardner, Stephanie, Born, Charles

Objective. To develop a diagnostic test for assessing cognitive skills related to metacognition in a physiology course.

Methods. Cognitive skills believed to be related to metacognition (visualizing lecture information and interpreting diagrams) were identified in a first-professional year (P1) physiology course and test items were constructed for each. Analyses included overall reliability, item discrimination, and variance comparisons of 4 groups to assess the effect of prior physiology coursework and diagnostic test score level on the first examination in physiology.

Results. Overall reliability was 0.83 (N = 78). Eighty percent of the test items discriminated positively. The average diagnostic test scores of students with or without a prior physiology course did not differ significantly. Students who scored above the class mean on the diagnostic test and who had taken a prior physiology course also had the highest average scores on the physiology examination.

Conclusion. The diagnostic test provided a measure of a limited number of skills related to metacognition, and preliminary data suggest that such skills are especially important in retaining information.

Keywords: metacognitive skills, assessment, learning, physiology

INTRODUCTION

When first-professional year pharmacy students fail examinations, traditional measures such as Pharmacy College Admission Test (PCAT) scores and overall undergraduate or science grade point average (GPA) often provide little insight into reasons for low achievement. Students who fail often have several things in common: (1) they do not monitor their learning, ie, they do not identify what they knowand do not knowbefore a test; (2) they spend inordinate amounts of time reviewing material they already know and not enough time studying information they do not know; and (3) they do not know if their study strategies are ''paying off'' until after an examination.

Metacognition is a term used to describe skills involved in monitoring learning and making changes in either how or what one studies. There are several common indicators of inadequate metacognitive skills. Text material that is heavily underlined or highlighted, suggests that a student does not know how to identify relevant information. For example, in a textbook used by students in this study, over a page is devoted to the cytoskeleton, including information about the size of each component.1 Students without adequate metacognitive skills could not use the course study guide to identify relevant information such as parts of the cytoskeleton and the function of and processes related to each. Another indicator of inadequate metacognitive skills is resorting to reciting textbook information verbatim when asked to describe a structure or process, suggesting that study strategies are limited to memorizing without understanding relationships. For example, reciting memorized information verbatim when asked to describe action potentials in skeletal, cardiac, and smooth muscle. Another indicator of inadequate metacognitive skills is a poor non-technical vocabulary. For example, students who do not know the meaning of the word sequestering, which is central to pharmacological mechanisms involving metals, cannot understand how such mechanisms work.

Since prior achievement does not always correlate with achievement in pharmacy school,2 among the first indicators that students do not have or use appropriate metacognitive skills are low scores on their first examinations. The objectives of this pilot study were to identify skills related to metacognition, construct a diagnostic test of these skills, and beta-test the diagnostic instrument by administering it to entering pharmacy students. This study was the first step in developing a valid and reliable diagnostic tool or test, called a Learning Skills Checkup which would serve as an early warning system for identifying students with poor metacognitive skills. Students who were identified as not having relevant metacognitive skills could then be counseled to attend workshops or individual sessions to help them develop these skills before the first round of tests. Although opportunities for developing these skills have been available to students at this institution for many years, generally such services are more widely used by students with good metacognitive skills than by students who are struggling.

The pilot study was designed to address 4 questions: (1) Since all items on each part of the test were intended to assess only one skill, did all items on each part of the test actually assess the same metacognitive or information processing skill? (2) Was the test reliable and did items discriminate between the students who did and did not have each skill? (3) Since the metacognitive skills test was based on physiology content and only some of the students had taken a physiology course prior to entering pharmacy school, were test results influenced by prior knowledge? (4) Did the diagnostic test do what it was designed to do: provide a measure of the impact of a variable called metacognitive skills on achievement? If so, higher scores on each part on the test should be reflected by higher scores on the criterion measure: the first examination in physiology.