House of Cards?

Software Magazine, Dec, 2000 by Janet Butler

Vulnerable e-business sites, many with lax operating discipline and guys like "Fred," place new demands on--and open new opportunity for--disaster recovery and business continuity providers.

E-commerce has spurred changes in most systems management disciplines, with disaster recovery (DR) and business continuity (BC) planning seeing their share of new requirements, emphases, and solutions, although traditional IT-level disaster recovery remains critical. One of the most difficult but intriguing differences in DR/BC for dot-coms versus traditional brick-and-mortar companies is in the current attitudes and behavior toward disaster recovery, which create new risks.

Consider, for example, this scenario. Say you're a venture capitalist (VC), debating whether to invest in an enticing new dot-com, which has poured more than $10 million--along with the requisite blood, sweat, and tears of its several hundred developers--into the development of its business. The site is pretty close to live, and the VC, wanting to know if it will stay up and perform well, has hired SunGard, Wayne, Pa., to check out its vulnerabilities.

The dot-com is doing internal failover, SunGard finds, but the backup servers are located in the same facility as the primary servers. Sure, that guards against hardware failure, but if the facility is not available, the site is out. Oops!

The dot-com is also regularly backing up transactions via the information capture feature of its Oracle database. So far so good. But what about the $10 million of content that drives the business? Not to worry; the company has that covered. It's doing once-a-week backup of content.

But What About Fred?

However, with time to market critical, and several hundred developers, losing a week's worth of development work could be very expensive. Furthermore, the chief developer, Fred, is taking home the streaming tapes in his backpack. Hmmm. And Fred is not even an employee; he's an independent consultant recruited from the dot-com's number one competitor. Uh oh! Wonder how loyal he is. Not only that, but Fred might be hard to get hold of in the event of problems, since he has neither a phone nor a home. Somewhat of a free spirit, he carries a pager, and sleeps in different friends' houses.

This is a true tale, claims Ken Smith, president of SunGard Planning Solutions. In the dot-com arenas, to put it mildly, he notes that few arc doing a rigorous job of backing up information--which includes not just the database, but the content, wherein lies a great deal of the organization's expenditure. For example, there's little doubt that if Amazon.com lost transaction data, it would be painful, so the company must keep track of every one of its transactions. However, it's the HTML or application code that contains the descriptions of books and pricing information. If Amazon lost this content, it wouldn't have a business. Traditional IT shops back up their database in real time, but they're rarely required to back up application code. However, the requirements are different for e-commerce application code, which changes rapidly.

As an example of new attitudes on another front, there's the sacrilegious dot-com view that 80% bug-free software is better if it's ready to go online now, than if it's 90% right in a month. But aren't bugs in computer programs "bad?" Won't they result in broken Web page links?

And then there's documentation. True, most companies do a lousy job of documentation, but brick-and-mortar organizations know they should do better. At least in large data centers every piece of equipment is bar coded, and documentation illustrates how it's all plugged together. However, even the largest dot-coms don't consider documentation to be any good, so it doesn't exist, and no one knows how the equipment is bolted together.

No Data Protection Umbrella

The lack of good operating procedures creates a "bit of a house of cards" for SunGard's Smith. This not only causes difficulties in communication, but also creates new areas of risk, since many things are no longer covered under the data protection umbrella. Without the standards and procedures of large IT organizations, he notes that Web sites can be down 18 hours over card frying (hardware failure): 12 minutes to get tip, and 17-plus hours to find and work out incompatible relationships in code. But shoring up vulnerabilities due to new attitudes is only one area of difference that e-commerce DR/BC providers must address.

Another is in the very definition of "disaster," which has changed from a natural emergency, such as a hurricane, to include man-made happenings, and, indeed, all forms of disruption to business, even "events." And there's no doubt that business disruption can be costly: Lloyd's of London, for example, reports that e-commerce companies lost more than $20 billion worldwide in 1999, due to computer outages, downtime, and hackers.

In addition, apathy once made disaster recovery a hard sell, with organizations traditionally loath to make large investments in solutions until they were burned by a disaster. But since e-business has made availability, outages, downtime, and thus business continuity highly visible, disaster recovery is worrisome to dot-com executives who have become receptive to learning about their vulnerabilities. Still, they operate at Internet speed, so time and mind share remain tough competitors, and "getting on their radar screen is a challenge," says SunGard's Smith.