The dumb disk is dead: viva the smart disk!

Computer Technology Review, April, 2004 by David Freund

Microprocessor performance has largely kept pace with Moore's Law, with performance doubling roughly every 18 months. However, disk performance has not improved at anywhere near the same pace, causing today's CPU to waste more time waiting for data to be read from disk into local memory. The additional microprocessor muscle has given most commercial applications only a partial boost, as predicted by Amdahl's law of balanced system performance.

The gap in raw performance between disk and the CPU/memory complex was huge when the first disks were designed and built 50 years ago, and their basic principles of operation have not changed. Disk capacity has been growing at a tremendous pace, about 60% per year, but the speed of data access has been growing at a slower pace. The average data-transfer rate, usually measured in megabytes per second (MB/s), has grown about 40% per year. The number of I/O operations per second (IOPS)--an important factor in the performance of most commercial applications--has grown at a mere 16% annual rate. That's a far cry from the 67% yearly improvement rate in CPU performance.

As a result, database systems and many other applications try hard to avoid disk I/O and perform as much work in memory as possible. In fact, this method of disk-I/O avoidance is the primary reason 64-bit processors have become the norm in high-end servers--each application or database process can use hundreds of gigabytes of main memory to keep a local copy of its data. Even disk-array markers like market leaders EMC, Hitachi, and IBM are racing to increase their cache sizes as quickly as memory density and pricing allow--the more disk I/O that can be avoided, the better the system performs.

With all of this work to avoid storing and retrieving data on them, why do we still use disk drives? Because the 50-year-old design remains the fastest, most cost-effective way to store changeable data persistently. For most desktop applications and many departmental and low-end enterprise applications, there's enough performance headroom to last several years. Of course, solid-state disks (pure-memory devices that simulate disk drives) provide blazing I/O performance, but they remain too costly to purchase and manage for most customers. Unless a break-through occurs in persistent storage, current performance trends in commercial data centers will force frequently accessed data to continue moving into memory devices, leaving disks to be used for "fixed-content" or "archival" data storage and to hold backup copies of memory-resident data. (Sound familiar? That's just how disk vendors currently compare Serial ATA drives to higher-performing SCSI or Fibre Channel disks.)

A new kind of device that's much faster, denser, and less expensive per MB than the current 50-year old design is needed. Some of the notions being kicked around research labs include tiny surface-mount disk-arrays on a card; nano-electrical mechanical systems (NEMS), which are like micro-machines in silicon; and newer types of flash memory. No practical products appear to be near the horizon, but the need grows ever more dire.

But merely replacing disks with something faster won't be enough--it would only move the bottleneck to the storage network. Fibre Channel throughput has improved fourfold in the last ten years, but latency improvement hasn't kept pace. Even if it did, storage-network performance improvement would still be dwarfed by the thirteen-fold improvement pace of CPU performance over the same period.

The key to performance improvement remains one that database developers already understand well: keeping data that's most needed as electrically close to the CPU as possible. The current trend toward networked storage has obvious cost and manageability advantages, but it also further separates processing from storage--only to transfer huge amounts of it over the network in order to find the "important stuff" that should be cached in server memory. What if, instead, some of that processing was performed much closer to the storage media?

This is the central idea of the "smart disk." Also known as "data-centric computing," this is a model of storage where all of the application intelligence associated with a set of data is located on the storage controller directly attached to the disk containing that data. It has the added advantage of lengthening the time available for a good replacement for today's disk drive to be developed.

A good first step in this direction is to off-load just a subset of functions from the main server--things like the detailed device control that has already been off-loaded to HBAs and NICs; transport-level protocols such as TCP/IP, iSCSI, and others are also heading in the same direction.

The more radical end goal advocated by some researchers is to move entire applications to the device controllers, using higher-level protocols such as CORBA, COM , Web Services/SOAP, RMI, or something else from the alphabet soup of distributed-programming acronyms to wire everything together. In effect, it's a more-extreme form of scale-out clustering.