The typical alert process for a typical system administrator goes like this:

Ring Ring (Mr. Typical SA rolls out of his bed and looks at the phone.) (Stage Crew: Hold up 3×3 foot sign of alert: Critical: “Foo.baz.com is DOWN!”) Mr. Typical SA: “Balls.” (Mr. T SA walks over to his computer and starts to troubleshoot why his website is down) …

The rest of the story you probably know. In summary, the SA diagnoses the problem and the site is back up in twenty minutes or so. He sends out an email explaining what happened and goes to bed feeling like a hero. If this sounds like you or your SA, congratulations, you have a competent system administrator who has just done is job right. No, really, I’m not being sarcastic, competent admins are hard to find. When I achieve competence I feel good about myself. Rightfully so, I think, everyone falls short from time to time and because this stuff is hard and doing your job well is a good thing.

Beyond Competence Alerts and monitoring are a big part of system administration, and it is worth trying to explore how to go beyond being a competent administrator when designing a monitoring system. So looking at the above story, here is what happens:

1. Alert: Tells the SA what has happened. Implemented by an automated monitoring system.
2. SA diagnoses problem: This is the why, and if you are one step ahead of me, you may have noticed that in the above, this is in no way automated.
3. SA fixes the problem.

Good system administrators enjoy automating their tasks. Figuring out what the problem is, or at least starting to is a task the begs to be automated. So if an alert can not only tell what has happened but also why it happened then a system administrator is going beyond competence by designing an intelligent alert.

Intelligent alerts that tell you “Why?” So what does it take for an alert that tells you why something has gone wrong? I think you need the following:

• Monitoring all the layers that make your application work. This means things such as the network, sql, processes, cpu, memory, hits per second, etc. This way when you get what I call the front-facing alerts they are accompanied by back-end alerts. The front-end alerts is What and the back-end alerts are Why.
• Log alerts are another key element. When something goes wrong admins generally check the logs as one of the first few steps. Log alerts can lead to skipping that step and since the alerts come at the same time and the automated temporal correlation can be very helpful.

The tools to accomplish this are generally well known. For alerts there is Nagios and all those related tools and things like Splunk and its alternatives can be used for the log alerting. What this really comes down to a lot of work. It is a big time investment to get all these alerts going. My first post on this blog regarding fault tree analysis can help you brainstorm what you need to monitor.

Predicting the Future The absolute best alerts will tell you about a problem before it actually happens. Probably the simplest example of this is to monitor disk space usage. When it has grown to certain percentage than you send alert. The admin gets the alerts before the server runs out of space.

Where it can get interesting is when you don’t have something as simple as a fixed value. An example of an alert I set up in the past for this is the growth of mail queues in Exchange. What you generally care about here is not the size of the queue but if the queue has been growing. To do this I used samples from a rrd file from past times and compared it to current checks to calculate the percentage of growth. The rrd file was generated from Nagios performance data that was recorded for graphing purposes.

Another example would be the amount of hits your web site is getting. What if you are getting a certain percentage of hits more for the time of the day, than you did the day before? Or what if you are getting a percentage of more hits for the time of day Monday that you did the last Monday? These sort of checks can predict a failure before it happens. Of course the big downside to predictive alerts is that is you don’t get credit for saving the day.