Monitoring vs. Logging

Enterprise Networks & Servers, Aug 2004 by Patterson, Michael

Network monitoring notifies us when something goes wrong unless, of course, a user calls first. Even though users often call support before the monitoring solution picks up the problem, network monitoring is still helpful, especially for keeping track of less important equipment and off hours notification.

In truth, monitoring today isn't just about failure, it's about trending. But when you get into trending you need two things - some means of graphing so you can detect emerging and past trends, and the experience to know what to look for.

Any seasoned IT professional that has been tasked with setting up a network/application monitoring solution will share with you that how you do it all depends on the needs of each specific business. Service providers, for example, have particular business needs that typically include the monitoring of numerous WAN links with detailed reports on such factors as uptime, utilization and latency.

Stores involved with perishable goods, on the other hand, have quite different requirements. They usually need to monitor temperature, humidity, smoke, water, etc. They also tend to demand 24/7 monitoring and wish to be immediately notified whenever something wrong is detected in the environment.

Call centers are another matter entirely when it comes to network monitoring. They generally care most about their CRM application, the server it runs on, and its responsiveness and availability to the dozens or hundreds of people taking calls and servicing customers. While they may be interested in other facets of monitoring, these are the bottom line needs that will make or break call center operations.

Monitoring, however, suffers from one serious drawback. It is generally a reactive activity. Some IT people try to overcome this shortfall by monitoring more variables in the hope that it will make things more proactive. Experience soon convinces them, though, that all this approach does is add to the work overload without providing what they really want - advance warning of problem issues.

So where are these warning signs exactly? How do you monitor them? What do they look like? Well, it really all depends on the type of business. But your log is an excellent place to start.

Often times, the best place to find the warnings you need is in the log. Whether it is the syslog, eventlog or SNMP traps, the system messages generally contain the warning signs and it has been this way for years. So if it's that easy, why hasn't everyone been pouring through these logs on a regular basis looking for "warnings?" Ask any Sys Admin and he will give you the answer with raised eyebrows and a slight smirk on his face "Yes, right. I've got time for that." That attitude, though, may well be condemning them to a reactive operating future with little hope of getting off the firefighting treadmill.

Not that Sys Admins are alone. It seems that most professions tend to gravitate to the more reactive approaches. I recently took my truck to a local garage for repair, for instance, and asked the owner "Why do you have so many cars on the lot? Aren't these people upset that you haven't fixed their car?" "No," he replied. "Most of them are waiting on a decision, or I was told to get to it whenever." He went on to explain. "Don't say whenever to me because whenever means whenever I get to it." He then pointed to a dusty vehicle at the back of the shop. "That truck has been there for over a year."

His operating basis was to handle the cars first whose owners were the demanding and who followed up the most. So I handed him my keys and said I needed my truck back today. He laughed. I called him that afternoon and again the next morning. I had my truck repaired in less than 24 hours. The point of this is that way of working tends to permeate many IT departments.

One effective way to phase out such a "He who screams the loudest" mode of operation is to pay more attention to your logs. The downside is that as operating systems and applications tend to be very chatty, you end up with an excessively large log of system changes. And the more garbage is saved, the harder it is to find what really needs to be observed. That is where a good syslog server comes in. Its job is to sift through the stuff and highlight what you really need to be aware of. But again, how you configure this out is very much dependent on the specific needs of the business, the hardware you are using, the applications running, and the quality of what is logged.

What's a Warning ra

While it isn't really all that difficult to figure out the important things to monitor for the business, it can be troublesome to come to terms with what a "warning" is, what you want to watch for and how to prevent unnecessary notifications.

The answer to most of these issues is to invest in a good syslog tool. If all you need is to be notified for a specific event or syslog, many freeware utilities exist. Check them out at www.freshmeat.net. If you need to trend how often a message is coming in and whether or not other systems are generating the same message, you need an application that empowers you with ad hoc querying capabilities. The better applications even provide you with historical trends and extend correlation charts. Examples include Logalot, MoniLog and Kiwi.