Hierarchical storage management - Storage Automation

Computer Technology Review, March, 2002 by Christine Chudnow

Hierarchical Storage Management (HSM) is an archiving approach that migrates less active data off primary servers and moves it to less expensive storage, while keeping it available to users. George Symons, Legato's vice president of product management and development, defined HSM as "a way to use multiple storage environments for a conceptual single file system. Admin-istrators want to be able to set policy and move files from primary storage to a secondary or tertiary storage based on those policies."

It's no surprise that IT administrators need this sort of approach. Users are creating monster data files and IT is buying storage to match, but in fact most storage is seriously underutilized: common estimates for Windows systems are at 30%, and 40% for UNIX environments. Not only is storage hard to use to capacity, organizations also need the performance of disk-based systems. Caught in this cycle of under provisioning and over-spending, companies need approaches that will keep files readily available for users, yet will help them stop spending money willy-nilly on expensive disk-based storage systems.

HSM can help alleviate over-provisioning and performance problems by automatically migrating files along a hierarchy of storage devices from most expensive to least--top tier, middle tier, and bottom tier. The top tier usually corresponds to fast, primary disk subsystems like RAID arrays. A secondary or middle tier can be less expensive disk, optical, or even fast tape, while tertiary storage is usually tape or optical. In this case, HSM is an addition to backup and recovery software--instead of making copies of data and storing them elsewhere, it moves the data from more expensive primary storage to secondary or tertiary storage. The move decisions are largely automated, set in advance according to the company's business rules and by the type of data.

HSM is different from backup. Instead of moving files, which requires users to request a restore. HSM keeps migrated files available by leaving stubs at their original locations on the primary storage. When a user issues a call for a file, the HSM application locates it on secondary storage and returns it transparently to the user. The HSM application can base migration thresholds and policies on several factors and combinations of factors including age, date last used, size, user, and application. HSM improves utilization rates because administrators can set capacity watermarks. When data amounts reach capacity, the HSM application selects data by policy and migrates it to assigned secondary or tertiary storage, leaving a file pointer or stub in place.

Algorithms usually control these threshold events, but organizations must first hammer out the data management policies to define the events. For example, what is "active" versus "inactive" data? (This is trickier than it looks.) Is an inactive file one that hasn't been used for three months? Six months? A year? Perhaps an Excel file is only used once a year at annual report time, but when that time rolls around that file had better be available. The HSM application should also have the ability to migrate data without significant manual intervention, which requires using predictive algorithms on file-based and/or block-based data. It must also make secondary data locations user-accessible, ensuring that the data is transparently, and immediately, available to the user. This is even more challenging for databases than for files, since HSM applications must separate and migrate database objects to secondary storage, while keeping them completely accessible to queries.

First developed in the mainframe world, HSM is quite successful in that environment. Nick Tabellion, CTO of Fujitsu Softech, was with IBM in the '90s working with the heavyweight MVS operating system and DFSMS, which provides extensive data management services for IBM mainframes. One of the DFSMS tools is DFSMShsm, a hierarchical storage manager providing backup, recovery, migration, and space management functions as well as disaster recovery. Two factors helped mainframe customers to achieve a 70-80% device utilization rate:

1 The HSM component was fully integrated into the MVS operating system. Tabellion believes that "you have to have your hooks in the OS to make an HSM product go." When the MVS OS was updated, so were the HSM hooks.

2 MVS enables storage pooling. This is the infamous "virtualization" concept in open systems--in the mainframe world, this actually works. Storage pooling allowed the HSM tool to treat multiple storage devices as single volumes, so it did not matter to a user where a file actually was. If the user needed the file, HSM automatically located and restored it.

By the mid-'90s, vendors began developing HSM on open systems. However, implementations proved complex and only partially successful, which gave a perfectly good concept a very bad name. Tivoli and other HSM developers were faced with the challenge of needing to write HSM applications to multiple operating systems. Quite often those file systems had multiple versions and were subject to fast-changing environments. When customers added HSM to these operating systems, they added a layer of complexity at the machine level and the management level. In addition, open systems did not have workable storage pools since their storage was direct-attached and subject to disk space and distance limitations. HSM activity could also bog down the network as it sent migrates and restores across the LAN. Customers threw up their collective hands, and developers went back to the drawing board.