A dangerous money Waster: The hidden costs and security risks of in-house backup programs can endanger data security and the bottom line - Internet

Computer Technology Review, Feb, 2003 by Phil Roussel

For quite a few years now I have worked extensively with members of the international Linux community, primarily networking professionals who are looking for methods to optimize backup while maintaining a high degree of security. From Europe to the United States to the Far East, there is a common mindset reflecting the freedom of open-source solutions. The dedication to the open-source concept, the willingness to share advances in operating systems, has of course led to widespread advancements in software.

But there are times when it seems that such enthusiasm for an idea can get in the way of considering other reasonable alternatives. I often encounter this when speaking with people about commercial backup software. Recently, while visiting a potential customer with a large, complex, heterogeneous system, it was somewhat astonishing to hear that they were primarily using tar-based homemade scripts to perform intricate network backups. Their main justification was the cost of commercial software.

With a little bit of analysis, it is possible to see clearly that "cost" is a relative issue and that it is not the only important factor related to "homemade" backup software.

Based on experiences and conversations with dozens of system admins, it's my contention that in-house scripts or programs are:

* Comparatively expensive

* Poor performers

* Dangerous to use in commercial activities

Network backup software is much more complex than it sometimes appears. The mission of backup software looks quite simple: Take a copy of data from a computer and write it onto a storage device. Although easy to say, this operation is considerably more difficult to accomplish reliably and consistently. A network backup must deal with every individual aspect of computing such as file systems, CPU architecture, OS, access rights, network protocol, SCSI chain, tape and library management and more. Each layer is complex to master as a standalone topic, and interoperability between all of them requires the most proficient development skills available in this industry.

The basic Linux tools look OK to do the job, but they aren't. A tar-based program using NFS for network capability is a widely used utility developed by system admins. But it suffers from numerous flaws that sometimes make it useless. Common defects can include restrictions like:

* One tape for one backup for one machine

* No catalog of backed-up data, i.e., the entire tape must be restored to retrieve an individual file

* Questionable reliability

Conclusion: This sort of program should be reserved for occasional and very limited needs.

Although writing software is fun, it doesn't make much sense from a budget point of view. Simple cost analysis shows that in-house software is too expensive.

Compiling a simple tar-based script for one machine that writes to a local tape should take no more than a day. But, then you must do some testing, improve tape management, refine the data selected for backup, do some more testing, etc. You can easily spend a week on this. Then you expand the solution to several other machines and do some more testing. Finally, after a couple of weeks of refining the process in your spare time, you probably have something useful. Everyone pats you on the back for this achievement. And of course, you must write an extensive description of your work, the procedures to follow for backup, restores and troubleshooting.

Follow-up is not very convenient, either: loading a new tape for each machine, making sure that the script will work as designed. That's an easy 15 minutes per machine per day, maybe an hour for only four machines. And once a month, on average, you will need to analyze/modify/adapt your script to follow the evolution of the machines to be backed up. This can easily take another half a day per machine.

The math is simple. Basic script design, pre-install testing, followed by regular tape changes, then modifying the script to reflect changes in the network... two or three hours a week become 20 or more a month, a couple of hundred hours per year.

Considering an average salary of $50,000-$60,000 per year for an experienced network professional, these non-productive hours add up quickly. By the end of the year, the in-house tar-based "cheap" solution is costing the company thousands of dollars just in workmanship. And this is a recurring cost, in addition to the money that will be spent trying to analyze and fix features that almost work.

Where is the logic of a recurrent, unnecessary annual expense for software that almost surely does not do the job adequately? The only advantage over commercial software is that homemade software is pain-free in terms of issuing a PO and spending the IT budget. The company actually spends the money, but it is buried in the IT salaries. No one sees it, an invisible expense that eats into company profits.

A better solution is obviously proven, dependable backup software with a tape-library. Purchasing and installing reliable backup (hardware and software) is much less than the "true" potential cost of handcrafted software. And total cost of ownership (TCO) will be a very small percentage of the TCO of a homemade, meticulously written software. The cost/performance ratio leans even more heavily toward commercial software when you consider benefits like technical support by network engineers and the possibility of automatic software updates.