Backing up moves dairy supplier ahead - HP Hood - Company Operations

Communications News, June, 2001 by Elizabeth M. Ferrarini

T-1 facilitates New England corporation's outsourced solution.

Today, HP Hood--one of New England's leading suppliers of dairy products since 1846--provides frozen deserts, citrus, nondairy and specialty food products throughout northeastern and midAtlantic states.

As late as 1997, HP Hood ran most of its operations on IBM AS/400s connected to dumb terminals. In 1998, the company--based in Chelsea, MA--began moving employees to Windows NT-based ethernet LANs. A WAN connects the six offices in New England, and two offices in New York. About 550 employees use a mix of servers distributed across the eight locations.

The six systems administrators located in five of the offices would fire off the daily incremental backup and weekly full backup for each server. Backing up more than 100 GB of storage onto eight digital access tape devices, one at each location, turned into a 15-month, expensive and time-consuming headache for the IT department.

William Moulton Jr., HP Hood's IT network manager, says, "We'd start at 8 p.m. to do a full backup of the 35 GB on the corporate server. When systems administrators came in the next morning, the backup would still be running. We'd have to kill it so employees could use the server without any latency issues."

Backing up Windows Exchange Servers using the backup software proved unreliable. "Many times, backups would fail and there wouldn't be anyone around at 3 a.m. to switch the tape," Moulton says. "At 8 a.m., we'd have to restart the backup just as employees are getting on the network," not an efficient way to handle the IT for a $500 million company.

A DIFFICULT TASK

Moulton put manual fail-safe procedures in place. Before the systems administrators could handle other tasks, they first had to verify that the backup ran on the servers assigned to them, with all findings going daily to Moulton. "It took each one an hour and 15 minutes each day to verify his servers," he says. "The task could take longer if he had trouble logging into a remote server."

Keeping track of the rotation of off-site backup tapes for each of the server locations also proved cumbersome. To make matters worse, Moulton says management did not understand how expensive these backup and disaster recovery procedures were.

Then, Moulton read about Waltham-based amerivault corp. (then called Recovery Solutions) offering a secure, online backup-and-restore service. Moulton put the service to the test free of charge. For two months, amerivault backed up one server at HP Hood's corporate office.

To justify spending about $45,000 a year for the services, Moulton analyzed HP Hood's backup expenses. Yearly tangible expenses included $20,000 for tapes, and $8,000 for the backup software licenses. The intangible expense included $81,000 in wasted staff time (nine hours per day at $35 an hour). Two forthcoming expenses included $28,000 to replace the eight DAT devices within two years, and $8,000 to upgrade the backup software.

Since the solution was implemented, system administrators have been able to start each day by checking their voice mail and e-mail, and making sure that servers and the network are up. "They schedule the backups, and the service provider does all the work," Moulton says. "We don't even have to prepare monthly rotation files, nor do we have to buy any tapes."

Vytal Vault, from VytalNet Inc., Toronto, Canada, provides the enabling Delta technology for amerivault to carry out a transparent batch backup of only the changes from the last backup. Kevin Harris, amerivault's director of technical operations; says, "First, we do a full backup of a customer's files. Then we create a mapping strategy in an index file. Each time a backup occurs, we look for files that have changes, and then locate those changed blocks, compress and encrypt them, and send them over the wire. Each tape backup copies the entire file. We'll backup just a comma, if that's the only change."

T-1 TRANSMITS BACKUPS

Systems administrators downloaded and installed the VytalVault agents on their workstations and on each server. The software provides the step-by-step procedures for scheduling an automated backup or a series of staggered backups.

HP Hood's backups now take anywhere from five minutes to 20 minutes, rather than several hours. Employees can use the servers as they are being backed up. Harris says that during the backup the server can take a slight performance hit.

To transmit the backup to amerivault's DS3, HP Hood uses its full T-1 Internet connection, which goes to a port outside of the company's firewall. HP Hood's files are stored live on SCSI-disks on two RAID subsystems--an LSI Logic MetaStor RAID and a Dell PowerVault--located in an off-site secure data center. Backup tapes get moved to an underground vault in a nearby state. HP Hood uses 128 Blowfish for encryption.

The new system has restored the system administrators' willingness to retrieve files from backup tapes. Previously, they would have to search the backup software's log to locate the tape, then load the tape in the DAT device, and then go back, and find the file and restore it.