Spam-Filter Testing Flawed, According to META Group; Evaluations of Spam-Filter Products Rely on Antiquated Testing Methods; META Group Identifies Best Practices

Business Wire, Jan 7, 2005

STAMFORD, Conn. -- As organizations continue to battle the proliferation of spam in the enterprise, the importance of choosing the right e-mail hygiene vendor becomes increasingly apparent. In recognizing the critical need for spam-blocking tools, organizations and publications have rushed to develop testing processes in order to segment the vendor landscape and evaluate product offerings. According to META Group (Nasdaq: METG), a leading provider of information technology (IT) research, advisory services, and strategic consulting, while the importance of testing cannot be overlooked, the type of testing in which many organizations have engaged has been flawed, at best.

"At root, the issue with many of the most common testing techniques is that they do not simulate real-world conditions or replicate real-time anti-spam abilities," said Matt Cain, senior vice president with META Group's Content & Collaboration Strategies service. "Simply put, current testing techniques do not allow organizations to accurately evaluate product capabilities. Invalid testing can, and will, lead to inappropriate vendor selection, an unhappy CIO, and more spam than anticipated."

META Group analysts warn, in particular, about testing performed on forwarded mail; specifically, they discredit evaluations that forward a large quantity of mail to a specific destination for filtering purposes. Such testing methods ignore the sophistication of header information and IP address manipulation, among other aspects.

These methods also do not allow for real-time message checks, including sender Internet Protocol (IP) validation tests. Finally, using this approach, anti-spam filters cannot glean intelligence from the SMTP transaction -- a critical data point for many detection techniques, including traffic-shaping technologies, sender reputation services, and e-mail authentication protocols.

"Spam-filter evaluations based on a mail-forwarding scenario will not provide testers with the information they seek about the accuracy of an anti-spam tool in the workplace," continued Cain. "Equally as ineffective as the forwarding approach are evaluations that focus exclusively on spam capture rates and false-positive generation. Testers must learn to look beyond the easy statistics and measure end-user satisfaction, ease of use, and operational control."

A best-practice approach to evaluating e-mail hygiene solutions, however, goes beyond these broader metrics. Smart testers, say META Group analysts, will segment the vendor landscape into the three major delivery models (hosted, appliances, and traditional software load) and compare accordingly. Testers must also recognize that, like many technologies, spam-filter products will not provide optimal performance in a plug-and-play situation. Instead, testers need to do the appropriate tuning, enabling recipients to set up block/allow lists to attain a more accurate picture of blocking effectiveness.

About META Group

META Group is a leading provider of information technology research, advisory services, and strategic consulting. Delivering objective and actionable guidance, META Group's experienced analysts and consultants are trusted advisors to IT and business executives around the world. Our unique collaborative models and dedicated customer service help clients be more efficient, effective, and timely in their use of IT to achieve their business goals. Visit metagroup.com for more details on our high-value approach.