Methods of Linux bare metal restore and disaster recovery

Computer Technology Review,  Sept-Oct, 2006  by Anthony Johnson

• E-mail
• Print
• Link

Most companies think they are prepared for disaster by backing up critical data, which they can quickly restore if a loss occurs. Few companies, however, are prepared for a catastrophe-total system failure-in which they lose all their data as well as the entire operating system, applications, settings, patches and more, reducing their system to "bare metal" and experiencing costly downtime.

The Cost of Downtime

Rebuilding even a single system from scratch can take a few hours or several days-time in which management, employees and customers have no data access. As a result there are no sales, no customer service, no product shipments, no supply chain management, no call centers ... no revenue.

How Much Does a Single Hour of Downtime Cost?

The Meta Group stated that, in 2002, system downtime cost American businesses an average of $1 million an hour. A survey done by Contingency Planning Research and published in the April 3, 2000 issue of InternetWeek details hourly downtime losses for several industries:

Industry                    Cost per hour

Brokerage operations        $6,450,000
Credit card authorization   $2,600,000
Package shipping services     $150,000
Catalog sales center           $90,000
Airline reservation center     $89,000

In today's 24x7 world, even if downtime is only $1,000 an hour, a three day shutdown would cost a company $72,000. At $10,000 an hour it would be $720,000. Few companies can survive such a hit. In fact, a chilling study from Contingency Planning and Management magazine discovered that 40 percent of companies that were shut down for three days failed within the following 36 months.

Starting Over

Recovering from a bare metal disaster is a highly complex operation that can take hours, days or even weeks to return the system to its prior state.

Common tasks include:

* Repartition hard drives

* Recreate volume groups and logical volumes

* Remake software RAID devices

* Remake file systems

* Reinstall the Linux OS from the original distribution media

* Recreate users and groups

* Reinstall hardware drivers

* Reinstall additional packages and third party software " Reinstall all software updates

* Reconfigure the network and firewall

* Redefine and reconfigure printers and other peripherals

* Reinstall and reconfigure web, FTP and email servers

* Reinstall backup software

* Reinstall database software

Linux System Recovery Tools?

Some UNIX administrators, who routinely use recovery tools like AIX "mksysb" and HP/UX "make_recovery," are reluctant to move to Linux because that OS has no similar standard tools for full-system recovery. In addition, most Linux distributions are fairly unsophisticated about how they install themselves. In most instances they use a single hard disk that is partitioned with a single root filesystem containing all system files as well as other applications and user data.

In more sophisticated configurations, an administrator might take advantage of performance, flexibility and availability benefits by spending considerable time reorganizing the system to use LVM, software RAID, enhanced filesystem types and other advanced techniques. Unfortunately, all this time-consuming and costly hard work, which may evolve over several years, must be repeated if the system crashes or new hardware arrives.

Fortunately, third party vendors are making tools available for Linux systems to ease this pain. Often referred to as "system recovery," "disaster recovery" or "bare-metal restore" tools, they are capable of restoring a typical Linux system to its previous hardware configuration. Currently there are two methods of accomplishing this: disk image recovery and file level recovery. Each has its own benefits and pitfalls.

Disk Image Recovery

A disk image backup program views the hard drive simply as a group of sectors. It backs up all sectors (not individual files) on the hard drive containing the data and creates an exact image. By operating on the disk sector level, the program backs up everything-system files, master boot record (MBA), partition tables and all user data-with no regard to their content or structure.

Some disk image backup tools, which are a bit more sophisticated, first back up the partition table information, then each partition independently as if it were a disk image. This adds a bit more flexibility by allowing IT to restore a single partition instead of the entire disk.

A disk image restore begins by booting the system from media (typically a CD-ROM) that includes a simple OS, device drivers for accessing the disks and backup media, and a utility for rewriting the data. If the entire disk was backed up, then the entire disk's data is restored. If individual partitions were backed up, then the disk partition table is first re-created and data is restored to each partition.

While in some instances this can provide the fastest means of backing up and restoring data, the method's inflexibility may cause considerable grief during a system restore. A disk image backup is like a block of ice that will only fit properly into its original container. If the "container" has changed (such as a different disk size, type or location), the data will no longer fit, and the entire backup is rendered unusable.