Tales from the patch patrol

Software Magazine, July, 1998 by Ann Harrison

Patching your software security holes is a full-time job. Here's how to organize your team, buy tools, monitor key Web sites, put pressure on your vendors, and manage your service providers.

Software patches are more important than ever for IS administrators who must monitor and repair security holes before they're exploited. Even the best software is only as good latest revision, and attackers aren't as its the same development schedule as vendors. Is your patch patrol up to the challenge?

Perhaps they will be caught unprepared like the IS staff at Silicon Investor. The heavily-trafficked finance discussion site was hammered by a series of attacks in January that targeted their networked Windows 95 and NT machines. Believing they had a secure network, administrators watched in horror as their frozen NT 4.0 Web servers continuously crashed and rebooted under a hacker attack known as the "Blue Screen of Death."

Fortunately for Silicon Investor, the attack was halted nine-and-a-half hours after it started when their upstream provider began filtering the rogue packets. An accompanying synflood attack, which attempted to overload servers with a barrage of requests, was squelched by altering registry entries.

Searching for the source of the attack and a defense they could install on their Web servers, Silicon Investor's IS team began checking security newsgroups for similar break-ins. Three days later, they located Jiva DeVoe, a senior systems administrator with DevWare Systems in Phoenix, Ariz., whose computer security Web site had been hit with a similar exploit. Now known as Bonk, Boink, or NewTear, the denial-of-service attack crashed servers by bombarding them with packet fragments that slipped past company firewalls. DeVoe pointed Silicon Investor to available hot fixes and a patch that Microsoft had just released to counter the exploit. "They hadn't had their patches up to date on their servers, so I told them which patches they needed to get," he recalls. Silicon Investor scrambled to install the patches. They decline to discuss the ordeal.

There's no shame in being felled by the Bonk attack. Even Microsoft, and security experts like DeVoe, were initially befuddled by it. DeVoe recalls that when his machines were first attacked by the Bonk exploit, he sent captured packets to Microsoft, which reviewed them for a week and couldn't figure them out. Impatient, DeVoe reproduced the attack by reengineering a version of Bonk's predecessor, Teardrop.c, and sent this information again to Microsoft and to two mailing lists. Posted on a Wednesday, DeVoe says he got a message from Microsoft the next day thanking him for the information but adding, "Don't expect a patch soon." Later that day, someone else posted the Bonk source code and Microsoft issued a patch the following Monday. "It was decent turnaround time," says DeVoe of the Microsoft patch. "But if I had a mission-critical system running NT, I wouldn't want to see it down all weekend."

The Silicon Investor attack should have been a warning to other patch patrols, but many weren't listening. In March, a new round of Bonk attacks swept education and government sites. This time, the exploit automatically queried a site's domain naming system for all the hosts on a network and then systematically attacked them all. While a patch had been available for months, many systems were still vulnerable, including those at NASA. That same month, Solaris systems at the Pentagon came under attack from an exploit for which a longstanding patch had not been installed. "It wasn't set up as a priority," recalls Rob Clyde, vice president of the security management unit at Axent Technologies.

Since some vendors issue fixes for a specific problem and others bundle them in service packs, it's difficult to estimate the number of patches released. But Richard Power, editorial director of the Computer Security Institute, notes that more than 100 security holes have been documented in NT since the operating system was first released in 1993.

Are company patch patrols falling down on the job? In many cases, the answer appears to be yes. But since firms don't want to draw attention to their security problems, most incidents involving unpatched vulnerabilities go unreported.

Gary Loveland, a partner at the Price Waterhouse Enterprise Systems Security Group, uses the latest exploits as a diagnostic tool to scan for unpatched security flaws in his clients' operating systems, firewalls, and applications. He says the majority of problems identified in security audits occur because systems administrators have not changed the default settings of newly acquired software or installed already available patches in existing systems in a timely manner. Loveland finds unpatched security holes in 80% to 90% of all the systems he audits.

Who's In Charge?

What's going on here? Maintaining adequate patches isn't rocket science. Yet many firms find themselves falling behind. Yours may be one of them. One of the biggest problems is that instead of organizing and funding a crack patch patrol team, companies try to get by with only one or two inexperienced sentries at the gate. "Budgets are usually the constraint here, not good intentions," says Ted Zlatanov, a systems administrator and leader of a patch patrol team at the surgical planning lab at Boston's Brigham and Women's Hospital.