At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

  • Monitoring allows you to send alerts if certain conditions are met,
  • It allows you to visualize trends in data,
  • Provides a method of assurance to the customer that their consumed services are guarded,
  • Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
A Large Network Operations Center

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.

 

One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.

 

Xzibit meme targeting monitoring systems

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

«
»
  • http://twitter.com/pcgeek86 Trevor Sullivan

    You gotta at least admit that, that “large NOC” photo is pretty sweet :)

    • Peter Grace

      If you like that one, check THIS out:  http://goo.gl/r3Lw9 — I’d love to visit one of these large telecom NOCs just to see if the stuff on their screen was even remotely important/relevant to what was going on in their company.

      • http://twitter.com/HipsterRouter Hipster Router

        comment redacted

  • http://profiles.google.com/jim.butts Jim Butts

    Where did you hear that WMI is going away?

    • Peter Grace

      You know, try as I might to find the blog I read a year or two ago, I was unable to find it again to provide the source.  I’ll strike that from the blog post as I cannot substantiate it now.

      • http://quux.tumblr.com/ quux

        WMI is not ‘going away in favor of PowerShell.’ Powershell has WMI-based cmdlets, for example: Get-Help Get-WmiObject.

        • Peter Grace

          Thanks for the clarification; I have left the comment in the post but striked, as is proper etiquette.

      • http://twitter.com/pcgeek86 Trevor Sullivan

        Thanks for fixing the blog post, but honestly your correction should read something more to the effect of: “WMI and PowerShell are completely separate technologies that serve different purposes.”

        They are complementary technologies to each other, and have almost zero overlap in terms of what they offer.

        • Peter Grace

          Thanks for your comment; after mulling it over I did add some additional information to the comment that should satisfy the need to seperate the technologies.  Thanks for your input.

  • http://profiles.google.com/francyci Chris Francy

    Why nagios-wsc instead of options like Check WMI Check WMI Plus?  Since you are a web company I know you probably have IIS already installed on most systems anyway, but why require an configuration on the monitored system when you don’t need it?

    • Peter Grace

      I’m a fan of nagios-wsc because it was the first one I used.  I’m not sure if check_wmi was around back when I started using it.  I haven’t used check_wmi to compare, but I’d assume their feature sets are identical.  Something I should have touched on in the article is the fact that we use Orion at Stack Exchange, not nagios.  This was meant as a more general introduction to monitoring than an expose of what we use internally.  Thanks for the comment!

  • sglez

    Nice status page. What software do you use for that?

    • Peter Grace

      Our developers wrote the page custom using data from our Orion databases.  All of the information is available via sql queries, but I don’t have the specific queries available to share at this point.

  • Jim

    Curious do you guys have the ability to switch to different time ranges in that monitoring ‘dashboard’? Also what’s the default period? Thanks!

    • Peter Grace

      On any of the graphs in that interface, if you click them you get an ajaxy popup of the chart with more detail, as well as the ability to change the timeperiod from the default of 1 day through to 1 year.  It’s pretty slick, I’m sad that our developers aren’t prepared to release the source code for it, as many people have contacted me asking for it.

      • Jim

        Thanks Peter. One more question – you’ve inspired me to start working on a dashboard for our systems. Are the graphs in your monitoring page inline images (rendered on the server) or generated via a clientside library (js, html5, svg)? or other?

        • Peter Grace

          The graphs are rendered via JavaScript on the clientside.  Nick Craver is writing a blog post about the monitoring environment in more detail; it should be available on blog.nickcraver.com in the coming days.

  • Pingback: Hello There « Nick Craver

  • sharegyan001

    If you are trading in NSE, BSE, MCX and in NCDEX then let sharegyan give you all stock trading gyan

  • Martin

    Hello, this is very interesting article. I’m wondering how do you get the current bandwidth of your linux load balancers. Correct me if i’m wrong i think you use haproxy on linux on them.

  • http://tshepang.net/ Tshepang Lekhonkhobe

    Regarding: “ My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.”

    I missed what the shortcoming was. Was it mentioned or did you miss it?