The workload driving the first industry-standard storage benchmark - Storage Networking

Computer Technology Review, March, 2002 by Roger Reich

This article is the third in an ongoing series exploring the evolution of performance analysis and benchmarking in the enterprise storage industry as well as the launch of the first industry standard benchmark for storage developed by the Storage Performance Council (SPC). This industry first will, in the next 18 months, establish a level playing field that will fuel a revolutionary landscape-of-comparison that will ultimately aid users, integrators, resellers, and vendors alike. This series of articles explores the foundation of this revolution, the detailed design of the SPC-1 benchmark, and how the benchmark can be used to affect more informed purchasing, integration, configuration, and tuning decisions.

The First Benchmark: SPC Benchmark- 1 (SPC-1) is the first industry standard storage benchmark and the first standard benchmark for Storage Area Networks (SANs). SPC-1 uses a highly efficient multi-platform and multi-threaded workload to emulate the precise characteristics of sophisticated enterprise class multi-user I/O applications. The SPC-1 benchmark enables companies to rapidly produce valid performance and price/ performance results using a variety of host platforms and storage network topologies.

Selecting the SPC-1 Workload

One of the principal challenges facing the SPC in building an industry benchmark standard was segmenting the enterprise storage market in order for an I/O profile (or workload) to be selected. Another challenge was to select a profile that represents a sufficiently broad segment of the market so end-users would find results relevant and useful in evaluating products. The SPC acknowledged that providing relevant storage benchmark data covering a breadth of enterprise applications (from seismic data process to banking) would no doubt require a set of benchmarks, but the question was, how many benchmarks? And could the set of benchmarks be minimized?

The solution rested with data collection and analysis by a number of SPC member companies. The needed elements of an I/O profile were agreed upon and a variety of application classes such as back-up, OLTP, print servers, email servers, etc. were targeted for data collection. Elements of the I/O profile included: read/write ratio, I/O size, locality, re-reference probability, sequentiality, interarrival time, and so on. This process took years, but ultimately the SPC collected a body of data that facilitated the accurate segmentation and targeting of the market for benchmarking as illustrated in Figure 1. This body of data pointed to the need for a small set of benchmarks that could in fact satisfy end-users' requirements, as outlined in the mission detailed for the SPC.

The SPC-1 Workload

The most prominent group of applications that displayed a "core" of common I/O characteristics was represented by OLTP systems, database systems, or mail server applications. This "core" of applications was characterized by predominantly random I/O operations requiring queries as well as update operations and simultaneously had threads of sequential I/O processing interspersed with random I/O operations. Because the focus of SPC-1 is on the commonalities of these applications, it was necessary to develop a model that would simplify the workload to the point that it highlighted the similarities of these business segments while removing any conflicts and details that weren't central to the narrow task of performance evaluation. In support, the model used in SPC-1 has two central scaling components: Business Scaling Units and Application Storage Units.

Application Storage Units (ASUs)

The SPC-1 benchmark synthesizes a community of users running against storage that is organized into the three logically separate Application Storage Units (ASUs) as would be encountered on a real-world application. The assignment of each ASU in SPC-1 is as follows:

* The Data Store (ASU-1) holds system-level data and represents the area where the server initially stores, for example, incoming mail messages or transactions. As users read their mail, it may remain on the data store, be transferred to the user store or be deleted. Forty-five percent of the capacity of the total capacity of the bench mark is contained in ASU-1. The Data Store has four parallel I/O streams associated with it. There is a read and write stream that is uniformly distributed over the entire address space, as well as some highly localized I/O to specific areas of the ASU. Additionally, there is a sequential read stream present. The I/O intensity for ASU-1 represents 59.6% of the total SPC-1 I/O command traffic.

* The User Store (ASU-2) holds user-level data and represents the area where, for example, users store their personal email, or a transactions manager stores information persistently in a database. Forty-five percent of the capacity of the total capacity of the benchmark is contained in ASU-2. There are three parallel I/O streams associated with ASU-2 User Store. Similar to the ASU-1 Data Store, the User Store also has read write streams that are randomly distributed across the entire address space of the ASU. There are also localized I/O streams, although there are fewer of these than are present on the Data Store. The I/O intensity for ASU-2 represents 12.3% of the total SPC-1 I/O command traffic.