Change management gets critical

Communications News,  Oct, 2001  by Lenny Liebmann

• E-mail
• Print
• Link

Network managers have to rethink how they implement change and support changes implemented by others.

Here's a shocking statistic for you. The biggest cause of e-business downtime isn't flaky routers or crashed hard drives. It isn't hackers and it isn't viruses. It's screw-ups by technical staff.

That's right. According to Gartner Group, 80% of all e-business downtime incidents are caused by problems in IT processes--with poorly executed changes being the leading culprit.

Actually, that shouldn't be news to anyone who's been watching the trade headlines lately. If you check out the follow-up stories on any of the big outages that have affected companies like eBay, Schwab and the New York Stock Exchange, you'll see that almost all of them have been attributed to software upgrades, hardware reconfiguration, or some other scheduled change task. That's not only bad news for the companies that are suffering costly outages due to problems in the change-management process--it's also bad news for IT departments whose credibility is starting to suffer because of so many self-inflicted gunshot wounds.

How did things get to this point? One reason is that the industry has become so good at infrastructure management. We've developed such effective solutions for monitoring hardware, and we've engineered so much redundancy into our critical environments that equipment failures no longer pose a serious threat to our internal and external e-business applications. The current crisis is, in part, a consequence of past successes.

It's also important to understand just how challenging change has become for technology teams. E-business applications now rely on an incredibly complex chain of elements, each of which must be in good working order--and well-behaved in relation to every other element in the end-to-end chain--for the whole thing to work. These elements include network hardware, servers running various operating systems, highly "componentized" software across multiple tiers, diverse types of Web content, security systems, storage devices and more. At any given moment, a technician in charge of one of these elements can execute a tweak to his or her little piece of the puzzle. That tweak can have tremendous unintended consequences.

In addition to having more elements that may be changed at any moment, change itself is happening more frequently. Business managers put tremendous pressure on technical teams to enable new services, post new content, add users and create links with outside parties. This get-it-done-yesterday attitude creates a disincentive to implement changes with the discipline required to avoid today's increasingly common change-related snafus.

Added to this volatile mix are a growing number of young technicians who were raised in a Web culture that's completely unfamiliar with historic change-management practices, IT departments that are already stretched too thin and an increasingly specialized tech community where communications between disciplines is minimal at best. The result is a surprisingly temperamental e-business stew, where one false move can cost you several hours of panicked firefighting.

What's the answer? First, technology managers need to confront the seriousness of the problem. We need to recognize that this is a new issue for us that we can't solve by doing business as usual. Next, we need to re-evaluate our current change-management practices and start to improve them.

Without getting into too much detail, there are two keys to improving our change-management practices. One improvement is to actually have codified, disciplined processes. Too many technicians still place too much trust in their own competence.

They often scoff at checklists and processes. But checklists and processes work. One of the greatest safety measures in commercial aviation was the checklist. Did pilots already know what was on those checklists? Sure--but the process reinforcement of the checklist had a significant impact on how consistently that knowledge was applied.

Another key improvement is to apply common change methodologies and mechanisms across the entire IT organization. Networking teams can't maintain their own little system for reconfiguring routers, while systems administrators have their own approach to installing operating systems patches. There are just too many interdependencies between elements in the end-to-end e-business environment for that to work anymore. If someone is going to load a bunch of new Java components on an application server, everybody better know about it, so they can watch to see how it affects their technology territory. And nobody should execute any new tweaks in their territory until it's clear that the environment as a whole has stabilized.

Of course, change management is a more profound discipline than can be properly covered in an 800-word column, but everyone, including network managers, has a stake in seeing that it is effectively practiced across the enterprise. If 80% of your lost sales were the result of salespeople saying something dumb to your customers, you'd think something was wrong with how the sales department was being run. That's how the rest of the business is going to start looking at us if we don't get our change-management act together. We'd better do it soon.