System administrators can sometimes seem like they are trying to get everyone down. Someone goes to them with a great idea, application, feature, or just something they just want to get done. When the sysadmin comes in, instead of being enthusiastic about the great idea, he starts talking about viruses, hacking, earthquakes, floods, fires, Vogon invasions, and all the things that can go wrong. Because of this, the natural reaction is to stick the sysadmin in the basement and stop inviting them to parties.

In order to work past this issue, it takes a little work from both sides of the table. Coworkers need to remember that disaster is part of a sysadmin’s job and that Murphy’s law is part of what they do. Part of a sysadmin’s job to think about this stuff all the time because we are expected to have backups and to be able to recover from small events to major disasters. Major disasters are uncommon but do happen, hence the importance of offsite backups at a minimum. Small disasters, such as power failures, router failures are actually quite common. When a sysadmin is doing a really good job, people don’t even know about some of the smaller failures.

It also falls on the sysadmin’s shoulders not get stuck in this gloom and doom world. We have remind ourselves not to get in the habit of saying No to everything because something could go wrong. This is important because the danger is that that people will try to start bypassing you or just not believing you when you talk about very real threats because they think you are crying wolf. It largely comes down to being mindful of the bad things that can happen while being enthusiastic about all the good things that can come from a new idea or feature.

It can be tough not be the guy in the street with the end of the world sign when in a way it is part of your job, but learning how to find the balance is one of the challenges of being a sysadmin.

Eating Our Vegetables

Kyle Brandt

Our CEO Joel Spolsky wrote a book about hiring programmers that is titled Smart & Gets Things Done. The idea behind the title is that when it comes down to it, you really just want to hire someone who is both smart and will get things done:

“People who are smart, but don’t get things done … would rather mull over something academic about a problem than ship on time”.

This applies equally in the area of system administration. Our CTO Jeff Atwood recently tweeted a reference to this:

When I read that I felt like he was talking to the sysadmin team (I hope he wasn’t — but the fact that I felt like he might have been is enough). The reason for this isn’t that we are not getting things done; I think we work hard and get a lot of things done. Rather, I think we have tendency to pick the interesting tasks over the uninteresting or tedious tasks.

Assuming as a system administrator you actually like what you do, I think tasks end up looking like the following:

So there are some interesting tasks that would be good to do, but may not need to be done. Then you have most tasks, which if you like your job, both need to be done and are interesting. Lastly, you have the tasks that are not interesting but do need to be done.

With a tendency to give too much weight to the interesting tasks or pieces of a project, that category on the right will grow over time. I like to think of keeping that category under control as eating my vegetables. Ideally, one maintains a good balanced diet all the time. But in reality there are times when you need to get on the scale and come to grips with the fact that you might need to go on a healthy diet for a while.

Lastly, don’t worry, the irony of taking some time to write this post is not lost on me.

This Saturday, August 6th starting at 3pm EDT (19:00 UTC) we will be taking our sites offline for maintenance. We are aiming to keep the downtime to about one to four hours. Should we need to extend this window I will post updates on this post.

During this window we are going to be patching our database servers. We are also migrating our servers to a new AD domain so we have everything under a consistent naming structure.

Hello, our name is Stack Exchange and we have an alerting problem. It hurts us, our friends, and our family. We are not sure how we got here. Sure, we get some extraneous alerts, but everyone does right? Then one day we woke up and had an inbox full of alerts. We wrote it off. We told ourselves that it happens to everyone. But then it happened again, and then again. We don’t want this. We don’t want to live like this anymore. We are ready to pick ourselves up. We are ready to face this problem and live a new, and better life.

Don’t know if you have a problem? Here are some of the signs:

  • You get alerts that you just don’t care about, because of this you maybe don’t see the ones you do care about.
  • The more serious ones wake us up in the middle of the night when they don’t need to because someone else is already dealing with them.
  • When something major happens your inbox is flooded.
  • You set up email rules to handle them.
  • You are ashamed.

If like us, you have an alerting problem and you have admitted it, I believe finding the righteous path starts with one rule:

Every Alert Requires Action

Every alert requires action. The problem we have right now at Stack Exchange is that alerts don’t require that we do anything. If we are to address our alerting problem — I believe this, more than anything, needs to be fixed. When I mean every alert, I do mean every alert. So what sort of actions can we take:

  • If it is real problem and you are dealing with it — acknowledge the alert.
  • If it is a false alert, acknowledge and adjust the threshold level
  • If the alert was a flood of alerts, acknowledge them and set up dependencies.

In order to do this we need a few things. We need a system that allows to effectively acknowledge alerts without too much friction. We need cooperation from everyone to use this system once it is in place. Lastly, we need to accept that we don’t have the power to cure our alerting problem. We can, however, through constant vigilance, get it under control.