Data Analysis by Walking Around - using student data in school administration

School Administrator, April, 2001 by Francis V. Barnes, Marilyn Miller

One district's unique process for assessing the impact of teaching on student outcomes

What happens when you walk around schools intent on capturing the voices of roughly 1,300 students--half the student body in your district--not once, but twice a year through structured personal conversations? What do students' voices add to our knowledge about instruction and learning?

Palisades School District, located in a rural area north of Philadelphia, has a vision of becoming a data-driven decision-making district.

Our vision of data worth collecting extends beyond the traditional realm of scores on standardized tests. It embodies a cross-section of students' voices that is both broad and deep, providing a window into how students approach learning and how well they think they have mastered material. Our task is unique, we believe, because we invite educators from other districts as well as our own staff to gather feedback--one student at a time--in personal one-on-one conversations.

Over the past 2 1/2 years, we walked through our buildings culling data from short personal conversations with students to gain a qualitative dimension reflecting skills students in our district were learning and what knowledge they were applying.

Even the higher-ups in the district are not exempt from the walk-through process. Both of us personally have handled one-on-one conversations with more than 100 students during each fall or spring set of interviews and we lead orientations for the interviewers.

We knew that qualitative comments would not replace the numbers we have traditionally collected and continue to collect. But we suspected that conversations would help us gain a context for those numbers and test scores. Students' voices could help us to see better why and where some students are having trouble. Furthermore, they might help us pinpoint where students who might otherwise have been overlooked were not being sufficiently challenged.

Unfiltered Evidence

What we learned were not all surprises; students' unfiltered comments of their work and understandings enlightened us, enthused us and, in some cases, alerted us to instructional and testing issues that needed schoolwide and/or districtwide attention. We have no doubt that their comments have had an impact that will, in time, translate into higher achievement. We now have evidence that the ideals contained in our school district's mission statement are being realized.

Take, for instance, the rich responses students gave us during our walkthroughs when they were asked to tell us how they make predictions while reading. We asked: "What clues do you use to make predictions when you read a story?"

From a 1st-grader, we heard: "[I] guess what it will be about by looking at the cover and pages until the end." From a 3rd-grader: "I guess and then check in the next chapter to see if I'm right." And from a 5th grader: "I ask myself questions about what I think will happen." Our walk-through process has generated valuable data for use by teachers and administrators.

After the walk-through each fall, our teachers analyze the data to determine what skills and understandings are secure for children and which areas require increased attention for the remainder of the year. Then in our spring walk-throughs we gauge whether students have improved in their understanding of the specific areas in which they had trouble in the fall. Teachers, working in small teams and individually, use the data to target their instruction.

We continue to rely on other means of measuring students' understanding of what they have mastered by administering standardized tests. This leads to test score numbers that, with growing regularity, are published by the news media, reported by state education departments and placed on school system Web sites as a means of documenting student learning and holding educators and schools accountable.

That type of accountability will remain with us. However, we believe the walk-through process is an excellent tool for student assessment. It also helps satisfy those who argue there is more to a student than a standardized test score.

We also are making better use of test data by using the Quality School Portfolio software pilot program sponsored by the AASA Center for Accountability Solutions and UCLA's National Center for Research on Evaluation, Standards and Student Testing, also known as CRESST. While this program enables us to see trends and patterns that help us to make instructional modifications, we find the actual voices of students to be another important means of determining student understanding.

Initial Forays

The walk-through process has evolved over the past 30 months into its present form. When we initially considered the purpose of the walk-through we talked with staff at the Institute for Learning at the University of Pittsburgh. They helped us define the purpose of the walk-through.

The walk-through focuses the participants on improving the core of educational practice. Richard Elmore, a professor of educational administration, planning and social policy at the Harvard Graduate School of Education, defines core educational practice as how teachers understand the nature of knowledge; the student's role in learning; and how these ideas about knowledge and learning manifest themselves in teaching and class work.