Preparing for worst-case scenarios in the data center: lessons learned - Disasters Recovery

Computer Technology Review, Sept, 2003 by Bob Maness

The recent Eastern Seaboard blackout, which cost New York City businesses more than $1 billion (or $36 million per hour) was yet another wakeup for a nation that has lived through a series of catastrophic events in its recent history. Operational downtime is costly to businesses and unacceptable to customers. Current heightened awareness of data center susceptibilities, coupled with today's demanding business environment, has spurred IT managers to begin addressing disaster recovery needs.

There are those who were prepared when the power went out, such as the e-commerce subsidiary of a major airline. Shutting down the company's e-commerce Website, which has more than 30 million unique visitors a day, for even a few minutes could deprive the airline of millions of dollars in revenue. To ensure continuity of business operations, the company's director of operations implemented an extensive disaster recovery setup nearly two years ago.

Designing the Data Center: Shatterproof Infrastructures

The airline's disaster recovery setup consists of two data centers, each running 20Tbytes of total disk space and 200 servers. The primary site uses VERITAS Volume Replicator host-based software (running on about eight different hosts) to synchronously mirror its data to a secondary site ten miles away. In addition, everything in both data centers is either clustered for failover, or hot clustered, using VERITAS Cluster Manager.

But what happens if the power goes out? To keep operations up and running, the two sites are located on three different power grids, with the primary site feeding into two separate grids and the secondary site on yet another. As we learned in the recent widespread blackout, separate power grids may not be enough. To provide further insurance, the primary site has two diesel generators and a number of different uninterrnptible power supply (UPS) systems. The secondary site has its own UPS system, as well as a natural gas backup system.

Planning for Survival: The Disaster Recovery Plan

Beyond the proper system infrastructure, it's also essential to have a plan. In fact, one-fourth of IT managers in a recent independent study of more than 850 IT professionals worldwide say that the very survival of their companies would be at risk if disaster were to strike and they did not have their disaster recovery plans in place. It is little wonder then that the study--which was conducted for VERITAS Software by Dynamic Markets, Ltd.--also found that one-third of U.S. respondents have been prompted by terrorism to create a disaster recovery plan.

Disaster recovery plans should always be accessible. They should also provide comprehensive coverage of system resources, including the entire data center, all of a company's remote offices, and both desktop and laptop computing environments. Many U.S. organizations today invest heavily in protecting data at the corporate headquarters, while leaving remote offices less protected, or possibly not protected at all. Also, research found that 34% of U.S. companies do not test their plans. It is essential that corporations begin to examine, test and update their disaster recovery plans on a regular basis.

Prepared data centers have every site completely backed up and running on a fully automated and redundant system, as well as a disaster recovery plan in place. With today's backup and recovery technologies easing burdens on busy administrators and limited budgets, critical data at all sites can be protected at all times, even the next time the lights go out.

www.veritas.com

Bob Maness is senior director of product marketing at VERITAS Software (Mountain View, CA)