Using chi-square and a PC to assess competency - laboratory management techniques

Medical Laboratory Observer, July, 2001 by Scott Warner

Here's how one hospital laboratory used the chi-square test and a simple Windows-based program their technologists' competency reading manual differentials.

Because laboratory medicine is more science than art, competency assessment must be rigorous and based on criteria derived from written guidelines. The idea is to tell people what to do and then watch them do it. If the steps are not followed, either the technologist or the procedure as written is wrong.

The Code of Federal Regulations [1] states that the laboratory technical consultant is responsible for evaluating and maintaining the competency of all testing personnel. This can be accomplished a number of ways including:

1. Direct observation of routine patient test performance, including patient preparation, if applicable, specimen handling, processing, and testing;

2. Monitoring the recording and reporting of test results;

3. Review of intermediate test results or worksheets, quality control records, proficiency testing results, and preventive maintenance records;

4. Direct observation of performance of instrument maintenance and function checks;

5 Assessment of test performance using previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; and

6. Assessment of problem solving skills.

The use of previously tested samples, which incorporate peer comparison, is the basis of a proficiency testing program. The manual differential, however, presents an interesting challenge. The method of counting 100 cells allows for a wide range of what are essentially subjective determinations. For example, with a finding of 10 lymphocytes, one can be 95% sure that the result lies somewhere between 4 to 18 (see Table 1, page 49). This can be an unsettling revelation to the many of us accustomed to modern precision, and this range doesn't narrow much until one counts 500 or more cells, which is hardly practical. These facts, along with the proliferation of automated differential analyzers, have moved laboratories away from reporting percentages and toward relying instead on absolute cell counts, which are a far more precise measure of the cell population. But the question remains, which differential is correct?

There's no doubt that how to read a routine manual differential accurately should be taught to new employees and students, but what happens in the real world is that a supervisor compares two differential counts and makes a judgment based on experience and intuition.

Is there a better way?

Maybe, according to Roy N. Barnett. In his book Clinical Laboratory [Statistics.sup.2], Dr. Barnett suggests using a chi square test to make this process more objective. Chi (pronounced as a hard "k" followed by a long "i")is the 22nd letter of the Greek alphabet. Chi-square, denoted [x.sup.2], is a statistical test that answers the question, "Is the difference between these two groups caused by something other than chance?" Statistical tests, such as chi-square, are designed to disprove the null hypothesis, which states that differences are due to chance alone. In other words, we assume that the differences are insignificant, or null, and try to prove they aren't. It's a kind of mathematical "innocent until proven guilty."

Barnett writes, "We can test agreement in white cell differential counts done by two different observers. Technologist A counts a smear and Job Applicant B counts the same smear." Barnett then applies the following formula, which omits the Yates correction:

[[chi].sup.2] = [[sigma].sup.n] [(a - b).sup.2]/a + b

where n = number of variables; a expected value; and b = observed value.

Results are referenced in Table 2, below, which lists 95% confidence limits for different degrees of freedom. Degrees of freedom (df) is loosely defined as the number of ways a series of data can vary independently. It answers the question, "How many items in the series are affected if I change one?" In this case, since changing one value potentially changes all others, the degree of freedom is n-1, where n is the number of cell types counted. For example, a manual differential count includes neutrophils, lymphocytes, monocytes, and eosinophils. Since we are counting 100 cells, changing one population means changing the other three, or 4 minus 1 (n-1).

In Barnett's use of the chi-square, if the value exceeds the 95% limit, the differentials are statistically different. Specifically, if there are three degrees of freedom and the chi-square value is greater than 7.81, the null hypothesis is disproved, and the differences are caused by something other than chance.

Barnett's methods have two major problems, which have been solved by technological advances since his book was written. The first is the lack of a standard. Which differential is correct? Who is to say that Technologist A is right? A teacher-to-student comparison is one thing, but in a hospital laboratory, trained professionals are often equally competent. Also, comparing one 100-cell differential to another doesn't eliminate the error inherent to such a small sample size. Fortunately, today's automated, 5-part differential analyzers count thousands of white cells, effectively eliminating the statistical variation caused by a small sampling. In other words, if the white cell count is 10,000, then the sample is 10,000 cells, not the 100 cells that would be counted in a manual differential. The standard then has become the automated differential.