In the spirit of storage - Cover Story

Communications News, August, 2003 by Drew Robb

Southern Wine & Spirits of America is one of the largest wine and spirits distributors in the country, with more than $4 billion in revenue and commanding 12% of the total domestic wine and spirits revenue. Operating in 10 states, it represents more than 300 suppliers and 5,000 brands, and services 125,000 retail and restaurant customers. Its 6,000 employees operate a distribution network that stretches from Miami to Maui.

While the company has gained a dominant position in its industry through its top-of-the-line warehousing and distribution network, its storage network was another matter entirely. Due to rapid organizational expansion, coupled with the explosive growth of storage consumption, server sprawl became the order of the day.

Networking personnel spent hours each day dealing with storage issues and backups became almost impossible to complete. This situation became most apparent with subsidiary Southern Wine & Spirits of California (SWS), the state where the company enjoyed its largest market share (more than 50%).

"Backup times were killing us, we were suffering serious amounts of downtime and having to add two or three servers each month just to keep up with storage demands," says Robert Madewell, director of networks at SWS.

SWS employs 2,000 people in California and maintains a distribution network comprised of executive, warehouse, transportation and training facilities. This network is managed from two sites--Union City in Northern California and Cerritos in Southern California.

TRAFFIC THROUGH TWO LOCATIONS

A total of 70 HP Proliant servers are installed between both locations, with traffic divided functionally rather than regionally (i.e., certain applications are run on servers at Union City and others are housed in Cerritos). Thus, depending on the application required-primarily Exchange Server for e-mail, proprietary collection software, a sales reporting system, SQL server databases, fax servers, Web servers and call center servers-network traffic was distributed accordingly using primarily Cisco networking gear.

Most servers ran Windows 2000, though some NT units remain to be upgraded. While the network itself could cope with the load, lack of adequate storage management often ground things to a halt.

"We were experiencing severe storage issues at both locations," says Madewell. "With storage consumption growing at a ferocious pace, coupled with no way to track usage, we would be faced with losing servers a couple of times a month due to them filling up."

Once full, users would be left with nowhere to store newly created documents. That led to lots of calls to IT, hours wasted manually correcting the problem and system downtime. According to Madewell, IT staff spent an average of two to three hours each day putting out storage-related fires and troubleshooting network storage issues--a big distraction for the eight staffers who had seen storage consumption grow fivefold without any more personnel being added to the unit.

In one year, for example, storage usage grew from 170 gigabytes to 400 gigabytes in the north and from 90 gigabytes to 200 gigabytes in the south. As well as lost time for users and networking staff, this growth also led to the addition of two or three servers each month just for new storage. Madewell also notes the addition of server racks, more power to server rooms, a larger AC unit and three more Cisco switches in the same year.

"We would spend hours pouring through volumes on the affected servers to see what space could be freed up," he says. "In many cases, they'd find large collections of PowerPoint presentations, MP3s, JPEGs, aging personal files, and multiple copies of the same documents residing in directories, as well as e-mail databases."

SMART FILTERING IMPORTANT

With no way to track storage usage centrally, two IT staff would go through the various directories, checking files one by one to determine which were of value and which could be deleted. One group of MP3s, for example, was almost deleted, until IT discovered the marketing department used the files frequently.

That near misstep demonstrated the need for smart filtering of stored files. "We would have been in big trouble if we had just eliminated those files," says Madewell. Meanwhile, SWS struggled to cope with backup demands.

A full backup using Veritas Backup Exec took from Friday at 6 p.m. until late on Sunday. Even a differential backup started at 6 p.m. could not be completed until 9 or 10 the next morning. With many users logging on at 6 or 7 a.m., open files (meaning they would not get backed up), slow backup and sluggish applications became the early morning norm.

"We really struggled to get any kind of a backup done," says Madewell. "We had to buy a lot of tapes, as well as two additional HP DLT tape drives to supplement our existing QualStar tape library."

SWS had been using Computer Associates (CA) software for years in its mainframe environment. So when CA organized a demo day onsite at SWS, Madewell made an appearance. One area of technology in particular made sense to him--storage resource management (SRM). He says one feature after another of CA's BrightStor SRM directly related to the daily storage nightmares he was experiencing.