File systems and storage - Special SAN Section

Computer Technology Review, July, 2003 by Christine Taylor Chudnow

Structured data storage gets most of the press--is this the realm of storage area networks, virtualization. automated provisioning. But this kind of well-behaved data only makes up 20% of enterprise data--an important 20%, but still a distinct minority. That leaves 80% of enterprise data as semi-structured (e-mail, combined database and file systems) and unstructured (word processing, spreadsheets, presentations, images). These files take up a large amount of storage capacity and can be difficult to manage, but they contain a lot of business-critical information. Jeff Erramouspe, president and CEO of Deepfile said, "There's a lot of attention paid to data center-centric data. There's very little information paid to the file system. That tends to be spreadsheets and PowerPoint files and Word files, or Access databases that are used by individuals or groups or departments. It could be creative data, things like source code if you looked at a software development company. This is a highly under-managed area."

File system-based storage management is evolving to meet the needs of unstructured data Dave Howard, president of Colorado Software Architects, decided to base their storage management development efforts on file systems because of its potential and opportunities. "If the file system is intelligent enough, any application will work with it. Frankly, it was fertile ground because not a lot had been done in that area."

Sun Microsystems has been firmly in this camp since their acquisition of SAM-FS, which archives files onto secondary media but keeps them online and immediately accessible. Suzan Szollar, Sun's product line manager in marketing, said, "In terms of direction, I think we're looking at all environment where we're distributing more and more of the file system across the SAN and the network, and also being able to work in heterogeneous environments."

Opportunities for file-based storage management exist throughout vertical markets. Some of the sectors with the highest file management needs include:

* Broadcast and video on demand

* Medical/healthcare

* Government/military/aerospace

* Education

* Oil and gas

* Manufacturing

* Life sciences

* Telco.

Erramouspe said, "You would never think of treating your structured dam the way you treat your unstructured data. There are five times as much unstructured as structured, but we rarely manage it. Everyone has DBAs, but who has file system managers?"

According to Marty Ward, director of product marketing, high availability and storage management at Veritas, file-based storage management development continues to build on traditional volume management and file system management services. During the 1990s, developers worked on cluster-based file storage management. such as integrating structured and unstructured data through clustered volume management services. This work flowed into today's file-based storage resource management (SRM) research, integrating file and block-based backup and recovery, and tiered storage.

SRM

Erramouspe said that even the largest companies haven't managed their files and file systems well. "The reason they haven't gotten it right is they're trying to manage the wrong thing. They're trying to manage disk and volumes and partitions, Our view is they need to manage their files, and if they manage their files well, everything else will take care of itself." SRM is one way of simplifying and extending management resources to file systems.

SRM sounds simple enough--a way of managing storage resources. However, the specifics differ wildly and range across physical and logical layers. All SRM packages monitor and report, some add automation and control features, and most are increasingly tied to application-specific service requirements.

Physical layer SRM: Works at the fabric and/or device Level. Example: Monitors and reports on port connections and alerts the storage administrator to network congestion or failure.

Logical layer SLIM: Manages stored data at the file level--files, file systems, volumes and volume groups (i.e., tracks and reports on the amount of space that an application's data has consumed).

Some logical level SRM packages already report on volume and disk usage and other broad categories, but file-based SRM adds file level reporting, Sample queries might include the one hundred fastest growing files on a network, the one hundred oldest files on an array, or one hundred files that haven't been accessed for over a month but aren't part of a critical dataset. This kind of detailed query allows administrators to make intelligent decisions about file archiving and retirement, and to identify orphaned files.

Backup and Recovery

Backup and recovery has traditionally been file-based, but has become more challenging because file volumes have grown so much larger. Chris Van Wagoner, director of product marketing at Comm Vault said, "The backup world in general, and Comm Vault in particular, has looked to block level technologies to provide better performance." Basing backup even on incremental file changes can be time consuming: backup applications must check each individual file for modifications since the last backup. At an average of a sixtieth of a second this does not take long, but when the backup application must consider 60 million files it becomes prohibitive to maintain a reasonable backup window. Instead, backup applications can speed up (and shrink backup window times) by looking at changed blocks instead.