Transparent capacity management

Computer Technology Review, August, 2004 by Jack Norris

Information Lifecycle Management, Storage Grids and On-Demand Computing are hot subjects in today's storage arena. The good news? These processes will help companies rein in their runaway storage environments by managing them better, faster and less expensively. The bad news? None of these strategies is here yet, and when they do get here most storage environments won't be prepared to support them.

However, the bad news is not necessarily inevitable. Companies can take steps now to adopt strategies such as ILM tomorrow, and those interim steps will immediately result in better, faster and less costly storage management. One of the important requirements for strategic storage management is the ability to distribute data among storage destinations without impacting end users or applications, and the inability to do this has been one of the major obstacles to optimizing storage environments. Modern IT departments struggle to manage data and storage resources, and must frequently add capacity and reconfigure throughout their networks, usually requiring downtime to do it (and sometimes corrupting data in the process). Transparent capacity management is the key to solving this issue.

Transparent capacity management allows IT administrators to perform management tasks without interrupting end-user access to data or applications, or having to depend on batch windows. Administrators can use transparent capacity management to dramatically increase storage utilization, effectively incorporate disk technologies like SATA, and optimize storage to meet compliance requirements. All of these features are available today, and also build the foundation for next-generation technologies like ILM and storage grids.

Capacity Management: Where it's Going

For most administrators, capacity management means constantly reviewing storage devices and working to forecast usage rates to make sure there is enough available storage until the next convenient time to take down the storage to add new disk. When an administrator does have to take storage offline, it typically requires extensive coordination with end users, late nights and weekends, and liberal amounts of coffee.

Unfortunately, the situation has gotten progressively worse as 24X7 requirements have put pressure on available downtime windows and compliance and business requirements have resulted in exploding data volumes. In these difficult and demanding environments, a manual capacity management process proves nearly useless.

Today, transparent capacity management is replacing manual capacity management as an important method of controlling storage volumes. Transparent capacity management adds storage capacity, loads data and manages large storage sets while working invisibly to end users and applications. Transparent capacity management allows administrators to distribute resources across multiple storage devices to increase utilization, eliminate over-provisioning, and ensure continuous access. This is particularly useful in environments like Web filer farms, where IT must efficiently allocate capacity to constantly changing data stores. Transparent capacity management enables them to distribute data in real time among separate drives, significantly increasing their farm's utilization and performance.

File System Virtualization

Transparent capacity management requires continuous data access while masking the underlying data location from end users and applications. Most importantly, transparent capacity management tasks must be executed while simultaneously allowing end users to access and update open files. File system virtualization provides both these capabilities and is the technical foundation for transparent capacity management.

The first approach to file system virtualization was to create a proprietary file system that had virtualization features within it. This provided benefits but didn't extend virtualization beyond its own proprietary file system. It also required a huge migration project to the proprietary file system before any benefits could accrue.

There are two subsequent approaches to provide transparent capacity management. The first is to create a proprietary switch, file system and namespace on a dedicated device and place that device between end-users and existing storage systems. This approach requires mount point changes and introduces a single point of failure and performance bottle-neck. The second approach is to provide file system virtualization by leveraging industry standards, integrating with global namespace solutions such as DFS, and relying on existing network switches such as Cisco Catalyst switches.

While both approaches enable transparent capacity management, the industry standard approach works within existing computing environments and does not require either mount point changes or software agent deployment on clients or servers. This architecture leverages existing network switches and performs capacity management by issuing standard file system calls and automatically synchronizing any changes across multiple storage destinations. At the same time, it monitors all client traffic and synchronizes it to source and destination storage resources. The file system virtualization technology maintains data integrity by avoiding collisions between client access and data movement, and only acknow-ledges the data movement to the client after it receives acknow-ledgements from both the source and destination storage resources.