Keeping Your Cool

Steve Murawski

I’m not talking about data center cooling here…

I was recently listening to The Ship Show podcast titled “Keep calm and PROD on”.  In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.

The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents.  The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.

The Scenarios

The hosts outlined a couple of scenarios for discussion.  They didn’t use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..).  In no particular order (going from memory and making up the classifications here), we have the “cowboy” response, the “neophyte” response, and the “deer-in-the-headlights” response.  I’ll provide a quick synopsis of these responses.

The “Cowboy” Response

This incident response archetype is very identifiable in many environments.  Picture it, the year is 2010, it’s 2 AM and your cell phone is blowing up with monitoring alerts.  The website is down and everyone is waking up.  You log in to your VPN and start gathering information.  Everything goes dark, and then everything comes back up, seemingly magically fixed.  It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.

This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I’ve observed over my years in IT.

The “Neophyte” Response

This incident response archetype is your typical “newbie”.  The neophyte does not have to be new to the field, just new to high pressure outage scenarios.  Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up.  The neophyte might not  be comfortable with getting a more senior engineer out of bed, or getting some developers involved.  Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.

The “Deer-In-The-Headlights” Response

The last one I remember the podcast covering was the “deer-in-the-headlights” response, where regardless of a person’s experience level, they just don’t respond well in a crisis.  Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders.  This leads to a fractured response, where people may be working at cross purposes.

What Should We Do?

What I’ve Learned In Previous Careers

Training To Be A Cop

You may not know this about me, but before I got into IT, I was training to be a cop.  I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns.  A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations.  (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)

In that training, we drilled how to respond to emergencies as individuals and as part of a team.  Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes.  Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions.  Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.

Part of the training regimen includes working past the failures and mistakes.  In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we’ve drilled in the classroom and on the range.  We are coached to get to cover, return fire, and call for backup.  We can’t dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.

Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits.  In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for.  We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera.  This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills.  Training helps minimize the “deer-in-the-headlights” and the “neophyte” responses and allows people who would experience those responses to fall back on training.

Working In A Public Safety Agency

I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department.  Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations.  I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.

Under the NIMS model, the first responding public safety officer is the incident commander.  As the situation develops, the incident commander role can change, based on who’s best suited to deal with the incident.  For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command.  (In my experience, this is after police officers have saved all the people in the building. 😉 ).  Let’s look at another scenario: officers respond to a report of a burglary in progress.  The first officer responding is the incident commander.   As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation.  Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.

NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer.  The PIO’s responsibility is to keep the stakeholders and public in general informed as to what is going on.  Part of the role is determining what information is helpful to share and what should not be disclosed.  The Safety Officer monitors conditions and ensures the safety of all incident personnel.  The Liaison Officer is responsible for dealing with all the coordinating agencies.  Defining these roles can help deal with the “cowboy” response.  If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus “just get x involved, he fixed it last time.”

How We Should Respond

These two experiences provide some basic thoughts into how we can approach incident response.

Drilling

Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn’t immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.

For operations personnel (developers, dba’s – yes them too – and sysadmins), this is critical.

For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel.  Guess what?  The knowledgeable guy isn’t always around when things go ill.

Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital.  Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally.  Quick – what do you do when your CDN stops serving content?  Have you drilled that scenario?  If you haven’t where do you even begin?

This is something we are going to be focusing more on here at Stack Exchange and I’m super excited about that.  We’ve decided that this is a priority for our organization and we’ll be dedicating time to this.

Roles

In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations.  The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team.  Certain situations we can just deal with and not need to involve others.  If I need backup (additional resources), I can request those, but if a situation escalates, it’s time to bring in more support.  We don’t have a strict command and control environment;  as we grow our technical staff I think that’ll be more defined.

When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience.  Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role.  We currently don’t have anything defined like that, though I hope as we start to drill these scenarios more, I’ll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.

In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be.  By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.

In our case, I can envision that as having our PIO designate one chat room  or google hangout as our internal status update location and during the incident he/she’d make regular updates to those areas.  In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.

As for the Incident Commander role, we’d need to train all of our on-call personnel, as well  the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident.  While we don’t have this defined yet, we had a short outage a few weeks back that illustrates how this can work.

  • About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom.  I wasn’t the on-call person, but I always monitor for severe external alert failures.
  • I rolled out of bed and ran stumbled down to my office and got online to start investigating.
  • First order of business, check our chat site.. oops! It’s down!  Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it’s running in New York with the rest of our infrastructure.
  • Next up, VPN..  which connects.. that means internet to our data center in NY is still good.
  • I jump in a Google Hangout (fortunately unfortunately for my co-workers,  I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team.
  • Chat comes back online.  I dropped a note in chat that I’m in the SRE hangout and troubleshooting.
  • I’m soon joined by Geoff Dalgas, one of our Core Q&A developers.
  • He and I discuss the situation and began validating the different bit of our infrastructure.
  • We determined that we were seeing an issue with keepalived on our load balancers.
  • Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
  • We also had several more developers pop in and see if they could offer any help.
  • We determined a course of action to remediate the problem and began to implement it.
  • Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.

In this situation, I was first on-scene and acted as incident commander.  If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.

Open Line

So, what do you do in your organization?  Do you drill and train for failures?  Do you test your backups?  Do you prepare your operations personnel for how to respond in an incident?

«
»
  • As we haven’t had the need for a 24/7 rapid-response crew until pretty recently, we’re still building the organizational readiness needed to do this well. We’re only just now getting enough operational experience for trends to show up, and there are a few areas we do need some work on.

    The big one is one you touched on: When something goes wrong, how does the first-responder notify others that the problem is being worked on?

    If the answer to that is, “IM/Call the engineer more knowledgeable about SystemX and then get into an ad-hoc Hangout with them,” that may get the problem fixed but doesn’t let anyone else know that incident resolution is already underway.

    If the answer to that is, “Drop a stream of updates into group-chat while hunting up that one Engineer for Hangout,” that’s much better and others can join in. Especially if you let them know a Hangout is underway.

    If the answer to that is, “Drop a stream of updates into group-chat, and then coordinate response through group-chat once the responding parties are all on-line,” that’s even better since you get a time-stamped log of who did what when, when did they knew it, and other such tasty details that’ll make writing the post-mortem even easier.

    Drills are a good idea on paper, but I have a hard time conceptualizing how they’d work in the absence of a completely redundant infrastructure.

    • Great comment! Drilling with out redundant infrastructure is tough.

      By redundant infrastructure, do you mean redundancy across your production environment or a duplicate environment to drill in?

      If you are expecting to have 24/7 operations and you don’t have redundancy in production, that’s one problem (and one that drilling in production will highlight).

      I don’t think a full replica of production is necessary for drilling, but could be handy.

      There are a couple of options. You can pick a “down-time” where you’ve scheduled an outage and use that time for drilling. The next approach is a bit more blatant, pick a point to test and do it. If you cause an outage, you’ve got a VERY realistic drill. 😉 Finally, you can replicate a lot of conditions in a virtualized environment or doing tabletop drills. Not as realistic as training in production or a production like environment, but it’s a starting point and I’d even suggest starting there before moving to a full on production or production-like environment.

    • manda

      Hello. Information was interesting. Thank.کرکره برقیراهبند اتوماتیک

  • gita

    was interesting. Thank..کرکره برقی

  • sayan

    Do this for all organizations can be implemented?اقامت ترکیهخرید ملک در ترکیه

  • mahtab
  • mahzad
  • sodabeh
  • sabz
  • Cedric Tejeda

    Interesting commentary – I loved the points – Does anyone know if my assistant might be able to get access to a fillable NY DTF IT-2 example to type on ?