Assessing Prospective Elementary School Teachers' Beliefs About Mathematics and Mathematics Learning: Rationale and Development of a Constructed-Response-Format Beliefs Survey

School Science and Mathematics, Feb 2004 by Ambrose, Rebecca, Clement, Lisa, Philipp, Randolph, Chauvot, Jennifer

An Example of Rubric Design

Using two different rubrics, one belief (Belief 6) about children's learning of mathematics was measured:

The ways children think about mathematics are generally different from the ways adults would expect them to think about mathematics. For example, real-world contexts support children's initial thinking whereas symbols do not.

Responses to Segment 8 about fractions were used to infer the respondents' support (or lack thereof) for Belief 6. (See Appendix F for the segment; see Appendix G for responses to items 8.1 and 8.2.) Specifically, we noted whether respondents recognized that children often misinterpret the size of fractions when they encounter symbolic representations and that contexts provide children with support in understanding the relative sizes of fractions (Mack, 1990). The greatest challenge in developing this rubric was to appropriately describe each of the three categories, particularly the middle category, that emerged from the data. We struggled to describe responses like that in Appendix G, which we had placed in the middle category.

In an early version of the rubric, we described the middle category in the following way: "Says Item d is easier than Item c but has a weak explanation." Group members realized that the term weak was insufficiently clear to describe this kind of explanation for future coders. In this instance, we assigned the response in Appendix G the middle score because of the ease with which the respondent thought a child could compare the fractions 1/5 and 1/8 in symbolic form. The respondent was insensitive to the challenge that symbolic representations can pose for children.

In another draft we focused on the respondent's claims about Item c, stating, "Says that Item d is easier but tends to think that c is either relatively straightforward OR would be difficult for reasons that are NOT related to the ways children typically approach the problem." This description was later revised to include also the reasons respondents gave for Item d being easier than Item c. The final version (see Appendix H) was more specific than the earlier versions and more focused on the degree of evidence the respondent provided for the belief. We devised at least six versions of this rubric, each more detailed and more focused than the previous one.

In rubric development, our first concern was validity. We asked ourselves the question, "What does this kind of response tell us about this belief?" Given the open-ended nature of our questions, sometimes respondents provided insightful responses that did not provide us with evidence about the belief in question. We were tempted to assign high scores to such answers, but we learned to look not at the overall quality of the response but rather for evidence of a specific belief. Our second concern was reliability; thus, we sought to develop rubric descriptions that would be clear for others using them. For this rubric, our coders ( 10 graduate students external to the project) achieved 87.5% interrater reliability (the target for interrater reliability is typically set at 80%).