Microsoft's DNS Woes Hold Lessons for Others. servers at Microsoft survive DoS attack

ENT, March 26, 2001 by Keith Ward

Microsoft experienced a series of failures of its Domain Name System (DNS) servers in late January, which caused a good deal of public embarrassment for the world's best-known software company. The experience led to a number of changes that experts say are long overdue--for Microsoft as well as other companies with similar vulnerabilities and poor DNS structures.

Microsoft says the first outages on Jan. 23 and 24 were due to a misconfigured router, which limited access to the DNS servers. The breakdown was followed on Jan. 25 by denial of service (DoS) attacks, in which hackers flooded the DNS servers with requests and blocked out legitimate traffic trying to get through. Most people could not access the numerous Microsoft sites for hours that day. Another DoS attack came the following day. Microsoft says that attack was less severe since it was ready for it. Outages totaled less than an hour on Jan. 26.

The DoS attacks should be an eye opener for the IT community, says Jim Magdych, security research manager at PGP Security. "DNS is critical to the infrastructure of the Internet, but it's often overlooked. People don't necessarily think about it first. It's just something that hasn't been as visible to the public at large," he says.

But should it have been obvious to Microsoft, a company with all its software experience and thousands of engineers? All of its DNS servers were on one subnet, a single point of failure that makes the likelihood of a successful DoS attack much greater.

Although not a good situation to be in, it is not unique, explains Michael Hoch, senior analyst at Aberdeen Group. He says Microsoft's practice wasn't shoddy. "The practice was common -- the majority of enterprises approach DNS [the same way]," he says. "Most have three to 10 name servers, and that's commonly how they set it up."

Still, Hoch agrees with Magdych: This common DNS practice leaves companies vulnerable. "DNS is almost like the autonomic nervous system -- it's essential but it happens automatically. DNS management needs to take a higher priority in the Internet today."

Magdych says DNS often grows haphazardly within a company, and that may have been what happened at Microsoft. "Maybe it's not the way I would've done it, but I'm not shocked it was done that way. If it's something that has grown over time, and just scaled up, network redundancy may not have been a top priority. Networks can be organic in their growth, and things just kind of expand. It certainly can be overlooked."

For its part, Microsoft says it took quick action to correct the problem. "As a follow-up... we had already begun the process of distributing access to DNS resolution across more than one network. We completed this task on Thursday [Jan. 25], providing an additional level of redundancy to our system," reads a statement from CIO Rick Devenuti on Microsoft's Web site.

That was a good first step, Magdych says. "It seems like a reasonable response to the attack they underwent. Anytime you're trying to strengthen your network, eliminating single points of failure is a key concept."

Hoch advises companies to "upload security patches as soon as you get them. These patches are readily available." Lest anyone believe this is a Microsoft-specific problem, it is not. The problem is actually worse with Linux, Hoch mentions, because of its open source nature. "Stuff [like the DoS attacks] happen with Linux all the time. Security patches are released almost daily."

When it comes to proper security, the operating system running the server, not just the DNS server, needs to have the latest patches and Service Packs applied, Magdych says.

Microsoft also instituted backup DNS servers at Akamai Technologies. In the event of a DoS attack or other failure, the backup DNS servers will automatically take over.

Akamai is an ideal company to host the backup name servers, given its wide distributed network. Akamai has about 8,000 servers spread around the globe, according to a company spokesman. The spokesman would not comment on its relationship with Microsoft.

One option for companies seeking to increase their security in this area is to have another company host all DNS responsibilities. There is one potentially big advantage to this approach, Hoch says: An outside company can use a proprietary system. "If it's proprietary, it's much harder to crack the code."

Magdych has some advice for companies that want to continue to host their own DNS. "Don't overlook critical sources, don't ignore security risks. For routers, [that means] basic things like leaving them unpassworded. That still, unbelievably, happens. And make sure basic filters are in place."

Do not forget to have backups on hand, too. "Backup all your zone files, all DNS records on the server, in case it's compromised or corrupted," Magdych says.

In Microsoft's case, all the security in the world will not make it bulletproof. As long as a network is attached to the out side world, there will be risks; and as long as a company is successful, it will be a tar get. But in the end, Magdych says the attacks could end up having a positive effect: "If this raises the awareness of the public at large on the risks of imprudent network architecture, maybe there's a silver lining."