Stack Exchange

We Have an Alerting Problem

Kyle Brandt

Hello, our name is Stack Exchange and we have an alerting problem. It hurts us, our friends, and our family. We are not sure how we got here. Sure, we get some extraneous alerts, but everyone does right? Then one day we woke up and had an inbox full of alerts. We wrote it off. We told ourselves that it happens to everyone. But then it happened again, and then again. We don’t want this. We don’t want to live like this anymore. We are ready to pick ourselves up. We are ready to face this problem and live a new, and better life.

Don’t know if you have a problem? Here are some of the signs:

  • You get alerts that you just don’t care about, because of this you maybe don’t see the ones you do care about.
  • The more serious ones wake us up in the middle of the night when they don’t need to because someone else is already dealing with them.
  • When something major happens your inbox is flooded.
  • You set up email rules to handle them.
  • You are ashamed.

If like us, you have an alerting problem and you have admitted it, I believe finding the righteous path starts with one rule:

Every Alert Requires Action

Every alert requires action. The problem we have right now at Stack Exchange is that alerts don’t require that we do anything. If we are to address our alerting problem — I believe this, more than anything, needs to be fixed. When I mean every alert, I do mean every alert. So what sort of actions can we take:

  • If it is real problem and you are dealing with it — acknowledge the alert.
  • If it is a false alert, acknowledge and adjust the threshold level
  • If the alert was a flood of alerts, acknowledge them and set up dependencies.

In order to do this we need a few things. We need a system that allows to effectively acknowledge alerts without too much friction. We need cooperation from everyone to use this system once it is in place. Lastly, we need to accept that we don’t have the power to cure our alerting problem. We can, however, through constant vigilance, get it under control.

Posted by Kyle Brandt (@kylembrandt) on August 3rd, 2011
Filed under Monitoring and Graphing
«
»
  • Grant Fritchey

    Excellent article. I agree 100%

  • Charles Hooper

    A client of mine is going through a similar situation. Our first step to resolving the alerting problem was a manifest much like yours: It was declared that all alerts must be actionable, clear, and contain enough information to take action. This is a good first step.

  • You need to join the @lusis:twitter and everyone else working on #monitoringsucks. These are the types of things that really just need to be addressed.

    https://github.com/organizations/monitoringsucks

  • voretaq7

    +1 and Amen my brother!

    No alerting system should send out messages that cause the recipient to go “That’s nice, now shut up and quit bothering me”.  Every alert should require action, and every alert should go to someone who can TAKE action (Alerting the California office manager that the basement in New York is flooded doesn’t help anyone).I would expand on your rules to define three additional characteristics of a good alerting system:It should alert me when we’re approaching a threshold (so I can reverse the trend).It should alert me when we’re about to cross the threshold (because my actions didn’t reverse the trend).It should alert me when we’ve blown past the threshold (Time to take emergency actions!).

  • Aaron Bertrand

    Great post Kyle! At my old gig we had a lot of alerts set up and they suffered the same types of conditions – one person didn’t know if the other person was also reacting (in fact several times people walked over each other’s changes), or one person assumed the other person was reacting and ignored it, or we left alerts repeating every day for conditions we’d long since agreed were benign. This is a good reminder for anyone who sets up or deals with alerts in any capacity, including folks who manage folks who do so – too many alerts can often explain burnout, and not enough alerts (or proper adjustment) can lead to disasters.

  • Oh lordie yes. Just last night I was brought out of bed by our storage array alerting me to the fact that the bad drive had finished striping out and was now ready for replacement. Of course the only part that fit into the SMS was the, “Bad drive! Alarm!” part. There were two alert messages since notices go out to two mail groups I’m a member of. So of course, one of the things I had to do last night was fire up my laptop and send a mail out to both groups saying, “this is an expected alarm, nothing to be concerned about, going back to bed now.”

    Had I not sent that mail, I’d have been woken up again three hours later when one of our night-owls saw the message and called me to make sure it was something I was 1: aware of, and 2: if there was anything they could do about it.

    Right now it’s just me and maybe one other person who defers to me in these matters, so the stepping on feet problem doesn’t exist quite yet. But it will. I don’t look forward to that.

    • voretaq7

      That’s a whole ‘nother blog post about on call rotations 😉

  • Rick Cusick

    Thanks for this – we’ve been enduring the occasional flood of false alerts at 3am recently.  You’ve inspired me to make some changes tomorrow to get it under control.

  • You need to break some IT managers, and a lot of systems engineers, of some cultural habits related to if they think the department is doing its job well or not.

    One common idea floated by management is to have red/yellow/green alerting, which I tend to find fairly pointless unless its defined in terms of actions like you’ve said.  What usually winds up happening, however, with “yellow” alerts is a bunch of ignored flapping yellow triggers for issues that aren’t really problems and don’t really require any action.  Simply turning those triggers off “feels” like you’re doing a worse job since you’re less aware of what is going on with your servers.  Arguing this with management can be tough, since management often views humans a bit like automatons that can’t, or shouldn’t, burn out on errant triggers.  On the other hand if yellow alerts are defined, in the monitoring system, as alerts which only go out during business hours or something like that, and really are associated with actions to take in order to prevent red alerts during the night, that could be useful — but I’d argue that ‘yellow alert’ is still the wrong way to talk about these alerts.

    Associated with this are the idea that triggers should be left on for things that are problems, but where there’s no action for the oncall to take.  For example, garbage collection may be getting high on a service, which may be causing flapping alerts, for which there’s no real action an oncall can take.  The service may need to be re-written to avoid GC pressure and that is probably sitting in some team’s sprint backlog.  This is technically a problem for the enterprise which ‘shouldn’t’ be ignored, in the name of doing our jobs better.  But leaving the alert triggers turned on just punishes the person getting woken up at 3AM in the morning, and unless you are in a very DevOps-ey environment, that person getting woken up at 3AM in the morning probably can’t do anything about fixing the alert the right way.  Even if you are in a DevOps-ey environment, the right thing to do for the enterprise may be to focus on other issues for the sprint and put the GC issues into the backlog and address them later, and its pointless to punish the oncall over this prioritization issue.

    It is definitely the right thing to do, though, to shut off alerts that have no associated action to take — but good luck changing the culture — I’ve been fighting that fight for a decade now.

  • http://airbrakeapp.com/pages/home (formerly Hoptoad) and http://newrelic.com/ both claim to have fairly intelligent alert handling built in. 

    • I realize that StackExchnge is probably too big to use these tools for everything, but I was suggesting that lots could be learned by trying it out on some small sample. 

  • Great post! We’re using Cacti but like you say, that ends up sending 50.000+ emails every day. Our administrators delete them with a mail rule. It’s a good thing those ISO certifications don’t demand that email is actually read 🙂

    I’ll be checking the blog for how you tackle this issue!

  • Great post! We’re using Cacti but like you say, that ends up sending 50.000+ emails every day. Our administrators delete them with a mail rule. It’s a good thing those ISO certifications don’t demand that email is actually read 🙂

    I’ll be checking the blog for how you tackle this issue!

  • Guest

    Yes, non-actionable alerts are a problem but mimicing alcoholism programs is not funny because your problem is trivial in comparision.

  • We (www.symplicity.com) used to have this issue, but solved it quite easily.

    We wrote our own monitoring system from the ground up.  One that makes sense.  One that aggregates alerts into a single e-mail when they happen in rapid succession.  One that only sends out texts on critical issues, and auto corrects/auto takes action on less critical ones.  You are correct in stating that every alert requires action.  The entire way our system was designed was to absolutely eliminate all alerts the monitoring system it self could auto-correct/fix.

    For example:  If Apache httpd on one of our cluster nodes crashes or is being flooded, we don’t need to be alerted about it.  Instead, we designed the system to just stop httpd, wait 5 seconds, and start it again.  If it continues for a prolonged period of time, or if it won’t come back up, then we’re alerted.

    The entire monitoring system is modular, allowing us to write modules that do different things, for example, a module for MySQL monitoring, a module for monitoring disk usage, a module for monitoring configurable server processes, and so on.

    We also took the time to write an intelligent web interface that quickly shows us the status of every machine we own, all with off the shelf jQuery/jQuery-ui and Raphel (www.raphaeljs.com).  Simply right click any machine on the website and you are presented with a list of options, ie: put it into maintenance mode where alerts and actions are ignored, restart a service running on the machine, get usage and history graphs for a number of metrics (and I don’t mean crappy rrd-tool graphs; pretty, colorful Javascript graphs), and so on.

    Mind you, our cluster is huge.  We run some fairly big systems, so this isn’t just a small-scale thing.  The data collection and monitoring it self is stored in a MongoDB database, and can scale to an infinite number of monitoring servers and backends, thanks to MongoDB’s sharding capabilities.

    I can’t even remember the last time we had a false alarm or text flood.

    Someday, hopefully this year, we hope to release a version of our monitoring system for public consumption on Github.  It is currently undergoing a third major re-write, with many new features and additions.  Once we feel this is ready, we’ll be sharing it with the world.  🙂

    • Leonid Mirsky

      Sounds great. What is the scale this monitoring system should deal with?

      From my experience, the solution can be great for a small scale, however when you try to implement it in a more robust environment, it can get very ugly…

  • Leonid Mirsky

    In this post the author mentioned the need of setting up dependencies, as a way of treating the “flood of alerts” case.

    As I am coming from the enterprise monitoring world, where this dependencies tracking solutions are available, can someone suggest an open source alternatives?

  • The company I work for ( http://www.pagerduty.com ) has a pretty alert-heavy culture, so I could be wrong, but one of the few good things about alerts that you don’t need to do anything about is that they can wake you up in time for the real alert — they just do that at the cost of waking you up far, far too often.

    I have an alert that’s set up to tell me something might be wrong with our outgoing emails, it’s only been right once so far.  But we knew ahead of time and we’re already logged in and versed about the problem — which is essential if you need to fix stuff right away.

    • Leonid Mirsky

      In my opinion it is very dangerous road. False alerts are having a tendency of becoming automatically snoozed. You may find yourself missing critical indications because sometimes these are might indicated a problem, however usually they aren’t.

  • We’ve taken a new approach for monitoring alerts by using a multi-condition rules engine. If you just alert off one metric/condition you’ll likely end up with spammy alerts, but if you can check for multiple conditions its unlikely the alert will fire unless its a real issue. We will also be supporting the running of custom scripts (alert handlers) so you can automatically action alerts without having to wake anyone up: http://blog.dataloop.io/2014/07/21/ifttt-if-this-then-that-for-devops/