Simplify complexity: the solution for improved service levels and reduced risk

Computer Technology Review,  Dec, 2004  by Dave Nocera

• E-mail
• Print
• Link

Complexity drives up support efforts, resulting in higher operational risk and costs and degraded service levels. In fact, research statistics often state that 80% of all unplanned downtime is attributed to people and process issues such as performing an operation out of sequence. But most times, we blame failure to manage networks or systems on people and process, not on the complexity itself.

Two means to manage complexity address improving Mean Time Between Failures (MTBF) and Mean Time to Repair (MTTR). Both are effective, but they also actually introduce more complexity into a system. For example, the hot-swappable disk improves MTTR but is internally far more complex than a non-swappable disk.

The primary vehicle to attack complexity is utilizing tools that automate tasks that are otherwise performed manually. These tools wrap technologies behind simple well-defined human interfaces. For example, some tools automate tasks for data center operations for Tier 1 support people. There are tremendous cost advantages to automating Tier 1 tasks because Tier 1 operations are generally staffed 24X7, so the cost justification for these "enterprise management tools" is amplified by the staffing costs of three shifts.

Not only do Tier 1 tools reduce the number of operational people, but they also reduce the skill level required because the Tier 1 tools enforce strict operational processes in a workflow. For example, an alarm event occurs in the net-work and sends a trap to the Fault Management System's operational console; the Tier 1 console operator detects the fault to be a failed hardware component on the network and dispatches a repair person opening a trouble ticket to track the event. The repair person receives the ticket and resolves the issue and closes the incident when the problem is resolved. The tasks performed by the console operator and repair person were clearly defined in a workflow that spanned the Fault Management and Trouble Ticketing systems.

Tier 1 tools automate operations while Tier 2 tools automate the operation of technology. In the simplest case, the Tier 2 tools automate the management of technology also by simplifying the tasks performed by Tier 2 support persons, such as configuring the technology. If incorrectly configured, extremely complex technologies like operating systems, switches or databases are unreliable. Tier 2 tools make these complex technologies manageable by humans.

A second class of tools for managing complexity affects MTTR by using redundancy mechanisms to route traffic around failures. Load balancers are a perfect example, in which traffic is routed around a failed web server to maintain availability in spite of the inevitable failure of the underlying technology.

Go back in time and compare Assembler language to Java, or MS-DOS to Windows. We can generalize that early technologies are simple in comparison to their modern counterparts, but many wouldn't know it by observing their crude controls. This is not only true for computing technology; this is true for all technology. Besides the manual adjustments for film advance, focus, aperture and shutter speeds, early photographers were also the director, artist, producer and developer of their own creations. In stark contrast, with the modern "point and click" camera even a child can take a great picture. During the exploratory stage, the controls for a technology are crude because the technology has not yet found its niche.

As technologies like cars, computers or cameras become widely adopted, the human interfaces become simpler but the technology "under the hood" gets far more complex. This is the first rule of simplification: the price tag for simplifying an interface is increasing the complexity behind the scenes. Tier 1 tools hide the complexity of the data center. An Ethernet port hides the complexity of the Internet just as an automotive ignition key is simpler to use, yet internally far more complex than its predecessor the crank-start.

Today's automobiles can practically think for them selves, adjusting fuel, brakes and traction, to compensate for changes in road conditions. If the family car breaks down today, Dad will call AAA, or even click OnStar. Pervasive technologies are designed to be operated by anyone, but the price for this utility is that experts must service them. It takes a small army of experts in most IT departments to manage the corporate computing assets and desktops to keep them humming along without interruptions.

Early technology requires experts. When a technology "crosses the chasm" (Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers by Geoffrey A. Moore, Regis McKenna) and becomes pervasive, the human interface is simplified while it encapsulates still greater complexity. Experts are then required to service the technology. TV repair, film development, VCR rental, gas stations and IT departments are examples of experts who offer services that support the automation of simplified technologies.