Building the first industry-standard storage benchmark - Storage Networking

Computer Technology Review, Jan, 2002 by Roger Reich

This article is the first in an ongoing series exploring the evolution of performance analysis and benchmarking in the enterprise storage industry as well as the Storage Performance Council's launch of the first industry-standard benchmark for network storage systems, the SPC Benchmark-1. In the next 18 months, this industry first will establish a level playing field that will fuel a revolutionary landscape of comparison which will ultimately aid users, integrators, resellers, and vendors alike. This article series will explore the foundation of this revolution, the detailed design of the SPC-1 benchmark and how the benchmark can be used to produce more informed purchasing, configuration, and tuning decisions.

Today, there are no rigorously agreed upon standards for the benchmarking and subsequent comparison of enterprise storage subsystems. A wealth of performance analysis tools, like the Intel developed IOMeter, exist that allow vendors and integrators to stress test storage configurations across a modest set of workload parameters such as read-to-write ratio and I/O request size. But until recently, consensus has not been achieved within the body of manufacturers who sell storage products on a "fair" benchmark to facilitate the "apples-to-apples" comparison of different products, technologies, and configurations. Perhaps the best-known de facto standard for the comparison of high end storage is the proprietary work done by Dr. H. Pat Artis of Performance Associates Inc. in the IBM OS390 benchmarking tool called the PAIO driver. It should be noted that there are a wealth of CPU, system, database, OLTP, and application-based benchmarks in the computer industry peddled by a variety of organizations from Standard Performance Evaluation Corporation (SPEC) to the Transaction Processing Performance Council (TPC), but the storage egg has never reached the frying pan until now. It should also be noted that while several "stress testing" tools exist for storage, none tackle the problem of producing scalable and correlated results in the presence of storage networks like today's Storage Area Networks (SANs).

The Industry Dilemma

Enterprise applications must be online 24x7 and be online with uniform/satisfactory response times, independent of minor variations in imposed load.

These response times, characteristic of a block engine, must be upheld independent of the presence of various data protection schemes, disaster tolerant configuration options, designs for sustaining fault-tolerance, cache sizes/options, and minor imposed application loads. In addition, with the advent of storage networks:

* Processors and storage are increasingly becoming independent purchasing decisions.

* Storage subsystems are becoming vastly more complex.

* An increasing variety of technology is available for building solutions (Fibre Channel, SCSI, iSCSI, adaptive RAID subsystems, etc.).

Despite these market conditions, the new "zero latency enterprise" lacks "apples-to-apples" performance and price/performance data to accurately compare products and technologies and configure storage solutions.

The Consumer Morass

An IT director faced with the acquisition of a storage subsystem for his/her open system is potentially faced with a list of performance facts and statistics that rival the tome of figures listed in the financial section of a modern financial newspaper. These include, to name only a few: disk RPM, disk seek time, disk data-rate, back-end bus data bandwidth, front-end bus data bandwidth, the number of front-end and back-end buses, cache size, protocol time, IOPS rate (for cache hits), as well as vendor unique and incomparable benchmark or stress testing results that are utterly incomparable to any other vendor's stress test results. In mass, these statistics represent as much help for an enterprise storage purchasing decision as the statistics in a modern financial newspaper represent for an attempt to beat the S&P 500. Individually, each of these statistics are as relevant to assessing the bottom-line performance of an enterprise class SAN-based storage solution in a real world environment as using a butterfly net to catch a hippopotamus.

The Epiphany

Five years ago, a phone call from a former college friend (of InfoSizing Corp.'s Francois Raab, who is the primary author of the now legendary TPC-C benchmark) produced the epiphany that started the Storage Performance Council (SPC). His proposal was simple: Apply the incredibly rigorous principles tot benchmarking developed at the Transaction Processing Performance Council (TPC) to storage systems. The idea was genius, less one dimension. Building a TPC-class benchmark that measured storage instead of applications/systems and produced comparable results across vendors and network storage configurations was a design problem tantamount to suicide. Fortunately, none of us who agreed to build the first industry-standard storage benchmark knew that we were in for five years of working nights and weekends. It's now clear why the effort to build SPC-1 took five years and why no other industry standards organization (despite prior attempts) has succeeded.