Storage Intensive Applications - Industry Trend or Event

Computer Technology Review, April, 2001 by Fred Moore

The unprecedented growth rates for digital storage applications have been well documented for the past few years. Annual storage growth rates can now be expected to range from 60% to over 100% annually for the next five years based on the prevailing economic conditions, the degree of effective storage management implementation, and the arrival rate of many new applications. At the heart of storage demand are numerous storage intensive applications.

Understanding storage-intensive applications means more than just quantifying high-capacity storage demands with variable performance requirements. Databases now represent an estimated 65 to 70 percent of all (block format) data on disk subsystems across enterprise, midrange, and distributed computing platforms. As these databases begin to scale beyond 10 terabytes in size, the need for timely information delivery accelerates. Application requirements for video-on-demand, HDTV, video mail, and electronic security are pushing the envelope on server capacity and transfer rate. Multimedia applications using video, voice, and text will soon drive throughput requirements beyond the 150-megabyte per second levels, but have relatively low I/O per second demands. At the other end of the application spectrum, data warehouse and OLTP applications can push I/O requirements beyond 25,000 I/Os per second, while transferring smaller data blocks. Projections for 100-terabyte applications requiring at least 100-gigabit (10 gigabyte) per second transfer rates by the year 2010 are now on the long-range planning horizon.

Sizing Storage Intensive Applications

The emergence of electronic medicine as a new discipline arises from the awareness that continuing advancements to the medical knowledge base on traditional paper-based methods is impossible. Recent studies suggest that 85 to 90 percent of all healthcare information is stored on paper or film. A typical radiological X-ray takes 12 megabytes of storage. If a hospital performs 200 X-rays per bed per year, a 500-bed hospital will generate 100,000 X-rays per year resulting in 1.2 terabytes of storage. Backing up this data doubles the storage requirement. A discharged patient's X-rays may seldom if ever be accessed again, but a lifelong archive of the data is still required. Others include CT-scans, digital echocardiograms and lab reports, and brain scans. Designing server, network, and data storage systems for new storage-intensive applications can quickly approach the limits for traditional file management and storage systems. The applications require careful planning or even outsourcing in order to meet servic e-level agreements.

The emergence of the World Wide Web represents a discontinuity in planning for storage growth. Here, the past cannot be used to predict the future. Internet applications that retrieve textual, e-business, audio, and video information are becoming almost impossible to control or plan for in terms of bandwidth and storage requirements. These applications regularly create and access files (as opposed to blocks) that range from 1 megabyte to several gigabytes or more in size. The Internet has already had and will continue to have a tremendous impact on bandwidth consumption and poses a new set of bandwidth management and storage management challenges with little help available in terms of measurement tools.

Internet traffic jams now pose a bandwidth challenge with an estimated 500 terabytes per month total volume in 1999. This does not yet include the potential impact of the wireless Internet market. The eventual impact of the Internet on the data-storage industry has not been determined. Generally unmanaged since its inception, e-mail was the first killer application for the Internet and now consumes several terabytes of disk storage at many customer locations. E-mail messages are often kept "forever" with little focus on deleting data that is no longer valuable.

A recent study by the Midrange Performance Group (http://www.mpginc.com) indicated that the average size of an e-mail message, including any attachments, has now exceeded 50 kilobytes. In addition, the average e-mail message presently takes an average of 17 hops to arrive at its destination creating a growing latency problem for Internet performance. Obviously, as e-mail growth explodes, storage management guidelines will become even more important, including the effective migration or deletion of infrequently referenced messages to lower, more cost-effective levels of storage. Soon, voice-mail and video-mail will join e-mail, pushing the requirements of storage and bandwidth far beyond current levels. What percentage of e-mail is junk mail? (Table 1).

Application Storage Profiles

As data continues to escalate in value, selecting the optimal storage system (disk or tape) that best matches the application's access characteristics is quickly becoming more critical. Applications demonstrate predominant access patterns over a period of time that characterize their normal profile. A read-intensive application may even demonstrate a high write level for shorter periods of time, though its predominant profile remains read intensive. Typical application profiles include read/write intensity, I/O (transaction) or throughput intensity, and random or sequential access patterns.