Best practices for implementing data lifecycle management solutions - Tape/Disk/Optical Storage

Computer Technology Review, July, 2003 by Albert Leung, Glenn Rhodes

What is the definition of high value data? One can argue that all data is mission critical at some point in its life. The idea behind data lifecycle management is that data has different values at different moments in time and should be managed differently in different phases of the life cycle. For instance, a law firm would consider the legal documents required for discovery and litigation in an active as very important before the final ruling is made, but afterwards these documents diminish in value over time. However, there are exceptions, such as the landmark case of Rowe vs. Wade, where the possibilities of re-use of the data are significantly higher.

Data lifecycle management solutions enable IT managers to assemble the appropriate combination of storage devices, media types, and network infrastructure to create a proper balance of performance, data accessibility, easy retrieval cost, and data reliability based on the relative value of two to the business. Consider how insurance companies store client forms on-line for two years to comply with government regulations and ensure responsive customer service; but after the retention period is passed, this data is migrated to off-line tape storage.

There are several components of a data lifecycle management solution. First, you need to have an inventory of your existing data and storage resources. You should classify your data and associate it with your business requirements and relative importance. Finally, you need to have data management policies that decide how to best match data with the appropriate storage resource. Should data be stored on-line, near-online, or off-line and when should data be replicated, migrated, or deleted? The ability to intelligently and dynamically automate these decisions and actions is the heart of a data lifecycle management solution. Following, are the best practices of implementing data lifecycle management solutions.

Everything Old is New Again

It is very easy to simply state that data lifecycle management is simply Hierarchical Storage Management (HSM) born again. Why invent this new category called data lifecycle management? Vendors first developed HSM products in the mid-1990s in the mainframe environment for distributed computing implementations. In HSM implementations, data automatically moves from expensive hard disks to less expensive optical media or to tape, according to specific policies.

HSM technologies have their shortcomings, though, which have slowed adoption. For instance, HSM products lack integration of data intelligence and scalability in a network environment, increasing management complexity in large disparate client/server environments. Before administrators can build HSM migration rules, they first need to understand the enterprise's data usage patterns and storage resource availability. What is your total capacity, available and used? What are your most active files and which files consume the most storage space? This type of information is typically not readily available unless a storage assessment was recently completed. Consequently, administrators using HSM products would often create migration rules in a vacuum. A rule such as migrate all *.doc files not accessed within the last 60 days may be appropriate fur one group of users, but may negatively impact productivity of another group or department.

Another issue stems from the fact that legacy HSM solutions were architected for static one-to-one relations between source and target volumes. Administrators were limited to manually configuring each dedicated HSM volume as a pre-defined source and/or destination device, and any migration rules could not be shared between multiple HSM servers. This has made HSM technologies impractical to implement for large distributed client/server environments. Finally, HSM technologies lack a global view of all storage resources in the network and instead require the administrator to constantly change the migration rules to adapt to the changes in the storage environment. For example, what if the original migration target is getting full? What if the administrator brings up a new storage device that can be used as migration target?

Data Lifecycle Management: The Need

Clearly, data lifecycle management at minimum represents a significant evolution of HSM techniques. IT managers are rethinking their migration strategies, fearing HSM was far too simplistic an approach. Today, many companies now want real-time access to their data for longer time periods. Consider how this requirement affects credit card transactions. In the past, credit card transactions were generally completed within a 120-day cycle. Because customers now have Web access to their credit card accounts, they now want the ability to review their transactions for the past year or perhaps longer. Even if customers don't use the data, it must be readily accessible or the value of the service is lost.

With the continuous reduction in the cost of DASD, IT administrators are storing more and more data on-line in order to speed access time and take advantage of new low-cost technologies such as ATA disk-based RAID. Customers want to implement tiered-storage solutions to automate data migration to these new low-cost storage devices. They want to migrate duplicate or inactive data from primary storage to secondary storage devices to reduce backup and recovery windows.