The 'Drill Down' Process - collection and usage of school administration data

School Administrator, April, 2001 by Philip A. Streifer

A strategy for asking the right questions and applying the data in useful ways

I thought I had prepared well, having thoroughly reviewed the school district report card published annually by our state education agency. I figured I could anticipate any question my school board might ask about our performance.

All was going well until one board member, who had been a reading specialist prior to retirement, picked up on an interesting combination of facts from among the hundreds of statistics compiled by the state about our schools and students. He observed a disparity between our two elementary schools. One appeared to be spending more time in language arts instruction than the other, and it also performed better on the language arts portion of the state's mastery test.

There it was, right in front of our noses, but we hadn't picked up on it. When asked to comment, all I could say was "We'll get back to you on it."

It was a couple of months more before I reported my findings to the board. The path I followed in figuring out what had occurred was time consuming and not at all apparent. In fact, the "answer" was quite surprising as it turned out that the school spending less time in language arts (and that had attained poorer test results) was actually "overperforming" given the entry language arts level of the school's students. Had I run with just the surface information displayed on the state report, I would have unfairly accused a school and its faculty of poor performance. The "drill-down" process that I engaged in, however, revealed a very different result.

Valuable Comparisons

The drill-down process starts with a global question or issue, which then is broken down into its component parts for analysis. Once the analysis is completed, all the data are considered from one's "helicopter view" to make a reasoned decision on next steps.

In the case of the state report card, the global question was "Is there a relationship between time allocated to learning and performance?" But the real question posed here by the school board was "Should we force the lower-performing school to add language arts instructional time?"

A good example of how the drilldown process works is one that many school district leaders have experienced in making a major purchase such as a home. On the surface the issue seems straightforward. We want to buy a home, which leads us to ask a series of questions: Where do I want or need to live? What is the cost of homes in one locality versus another? How much can I afford? What are interest rates and are they stable? How much are taxes? How much are the condominium fees?

As these factors are narrowed down, I can begin looking at various properties to make comparisons. Then, once I have selected one or two very good prospects, I can consider making an offer to buy. But how much should I bid given market conditions and potentially rising interest rates? If I bid too low, dragging the purchase process out, I risk losing that amount in rising interest rates over the term of the mortgage. Or I risk losing the home altogether, having then to settle for a higher-priced or same-priced home or a less desirable alternative.

Thus, I need to conduct a cost-benefit risk analysis to continue the process. A final decision or determination requires what I call a helicopter view--an overview and consideration of all the facts that can be gathered and analyzed in a reasonable period of time.

Statistical Differences

In the case of the language arts performance at the two schools, I started by simply verifying all of the information. Next I wanted to know if the schools were somehow using different instructional techniques. I didn't think so, but it was worth checking. Then I reviewed the experience of the staff--perhaps one school had a young, less-experienced staff. That proved to be a dead end as the staff was well balanced. Then I decided to disaggregate the data by student mobility rates, gender and special education enrollment. No luck there either.

A next logical step was to see if the mastery test scores between schools, as reported by the state, were statistically different. The score reported by the state was the percentage of students attaining mastery. But I wanted to know if there was a practical difference in scores between the schools.

To find out I entered each student's score into a spreadsheet and ran a Test--a powerful statistical technique to determine if group scores are significantly different. The analysis showed no statistical difference between schools. This was surprising. I wondered how these two schools, one with more students attaining mastery than the other, could have no statistical difference between their scores?

The answer, it turns out, is critical and lies in the nature of the measures reported by the state. Percent mastery is a simple "frequency," based on a cut-off score, while the T-Test is a more powerful statistic designed to help one better understand the nature of and the differences in the data.