Pinpointing a Slowdown: Historical Trending of Resources

Enterprise Networks & Servers, Apr 2004 by Patterson, Michael

Why is it so slow? is probably the most common question asked of IT. Unfortunately, it can often turn out to be one of the most difficult to answer. Why? Finding the reason for slows typically involves considerable investigation, and a whole lot of trial and error.

After all, there can be a myriad reasons for a slowdown. Is it the network? Is it the server? Is it the application? Let's assume you narrow it down to the network. While that simplifies things somewhat, it also creates yet more questions. Is it one of the routers or perhaps the Internet service provider (ISP)? Maybe it's a switch, or a problem at the end user's PC not to mention one of the many links that must be traversed to complete the connection between points A and B.

That's why when you go to find out, you absolutely have to have reliable historical data about the cause of sluggish performance. Regardless of whether the bug is network, server or application related, you are floundering around in a dark and lonely place if you don't have easy access to trend information.

The reason for this is simple. Most performance issues disappear as quickly as they show up. Someone experienced a problem earlier that day, you check your screens and everything seems to be fine right now. You are none the wiser, and more importantly, don't know if there is an underlying situation that could cause the same situation to happen again. Or worse, you may have just missed an early warning sign to a much more challenging network or system event that could cause much later heartbreak. It takes historical trending to solve this dilemma.

Historical Trending

Many of you will be aware of an excellent trending tool known as MRTG (Multi Router Traffic Grapher). It contains a Perl script using SNMP (Simple Network Management Protocol) to read traffic counters on routers and a C program to log and graph the traffic. These graphs are embedded in Web pages and show traffic over a period of one day, seven days, five weeks or twelve months. Although originally created to monitor router traffic, it will also pull and graph any other SNMP data from any network device.

MRTG is now used by numerous companies and organizations worldwide including NASA, Sandia National Laboratories and IBM Global Services. In addition, MRTG has been incorporated into several other open source network management products.

The MRTG website (www.mrtg.com) contains documentation, sample applications, and downloads of the software for Linux, Windows and Unix. Kudos to Tobi Oetiker, David Rand and the Global Community for making this software freely available on the Internet.

I'm sure many of you will have used it at one time or another to graph SNMP statistics from routers and switches. But how many use it beyond SNMP? For example, did you know you can perform transaction monitoring of an SQL database via an ODBC (Open Database Connectivity) connection with MRTG? And did you know that you can use MRTG to monitor many different statistics on a Microsoft server using WMI (Windows Management Instrumentation)?

Why would you want to monitor a server or application if you are the network guy? Because, until the IT team isolates the exact problem, the resources you are responsible for managing continue to be regarded with suspicion until the issue is fully resolved. And if people can't resolve the issue, they may take the easy route and just blame the network.

In most cases, it may be enough to use MRTG to graph CPU, memory, hard drive space, and user connections for your servers. These are all extremely useful statistics you can monitor via SNMP. But if you want to go deeper, you can also use MRTG to dig into application performance. For example, is the slowdown being caused by the SQL query or the configuration of the database itself?

Alternatively, it could be happening due to the fact that the application was programmed in the wrong language.

On the mission critical application servers, you should be monitoring traditional vital signs such as CPU, RAM consumption, hard drive space, network interface card utilization, as well as ping response time when possible. Should you decide to take it to another level of monitoring, you may want to consider monitoring the number of user connections to see if they increase or decrease prior to when a problem occurred. This would be an easy statistic to notify on as well. The same thing sometimes holds true for monitoring processes; a great deal of them are spun off prior to a server locking up. Disk reads/writes is another good statistic to watch; you can learn a lot about an application by comparing this statistic between servers. Look for applications on one server that arc talking to the hard drive far more than other applications on separate servers.

Finally, bear in mind that it generally isn't your responsibility to fix a vendor's software application. However, it is your job to help where you can and to be proactive where possible. In some cases, you may be able to utilize tools such as MRTG to help a developer identify some of the underlying issues with software. In other words, if you can see a problem coming or help the software vendor by identifying an anomaly, you are that much closer to getting the problem resolved.