Let’s talk about encryption. Specifically, HTTPS encryption. If you’ve been following any of the U.S. election debates, encryption is a topic that the politicians want to talk about – but not in the way that most of us would like. And it’s not just exclusive to the U.S. – the U.K. is proposing banning encrypted services, Australia is similar. If you’re really into it, you can get information about most countries cryptography laws.
But one thing is very clear – if your traffic is not encrypted, it’s almost certainly being watched and monitored by someone in a government somewhere – this is the well publicised reason behind governments opposing widespread encryption. The NSA’s PRISM program is the most well known, which is also contributed to by the British and Australian intelligence agencies.
The premise is simple:
- Provide free encryption certificates
- Make renewing certificates and installing them on your systems easy
- Keep the certificates secure by installing them properly and issuing them best practices
- Be transparent. Issued and revoked certificates are publically auditable
- Be open. Make a platform and a standard that anyone can use and build on.
- Benefit the internet through cooperation – don’t let one body control access to the service
Let’s Encrypt explain this elegantly themselves:
The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention.
The process goes a bit like this:
- Get your web server up and running, as per normal, on HTTP.
- Install the appropriate Let’s Encrypt tool for your platform. Currently there is ACME protocol support for:
- Run the tool. It will generate a Certificate Signing Request for your domain, submit it to Let’s Encrypt, and then give you options for validating the ownership of your domain. The easiest method of validating ownership is one that the tool can do automatically, which is creating a file with a pre-determined, random file name, that the Let’s Encrypt server can then validate
- The tool then receives the valid certificate from the Let’s Encrypt Certificate Authority and installs it onto your systems, and configures your web server to use the certificate
- You need to renew the certificate in fewer than 90 days – so you then need to set up a scheduled task (cron job for Linux, scheduled task for Windows) to execute the renewal command for your platform (see your tool’s documentation for this).
And that’s it. No copy/pasting your CSR into poorly built web interfaces, or waiting for the email to confirm the certificate to come through, or hand-building PEM files with certificate chains. No faxing documents to numbers in foreign countries. No panicking at the last minute because you forgot to renew your certificate. Free, unencumbered, automatically renewed, SSL certificates for life.
Who Let’s Encrypt is for
- People running their own web servers.
- You could be small businesses running Windows SBS server
- You could be a startup offering a Software as a Service platform
- You could be a local hackerspace running a forum
- You could be a highschool student with a website about making clocks
- People with a registered, publically accessible domain name
- Let’s Encrypt requires some form of domain name validation, whether it be a file it can probe over HTTP to verify your ownership of the domain name, or creating a DNS record it can verify
- Certificate Authorities no longer issue certificates for “made-up” internal domain names or reserved IP addresses
Who Let’s Encrypt is not for
- Anyone on shared web hosting
- Let’s Encrypt requires the input of the server operator. If you are not running your own web server, then this isn’t for you.
- Anyone who wants to keep the existence of their certificates a secret
- Every certificate issued by Let’s Encrypt is publically auditable, which means that if you don’t want anyone to know that you have a server on a given domain, then don’t use Let’s Encrypt
- If you have sensitive server names (such as finance.corp.example.com), even though it’s firewalled, you might not want to use Let’s Encrypt
- Anyone who needs a wildcard certificate
- Let’s Encrypt does not issue wildcard certificates. They don’t need to – they offer unlimited certificates, and you can even specify multiple Subject Alternative Names on your certificate signing request
- However, you may still need a wildcard if:
- You have a lot of domains and can’t use SNI (I’m looking at you, Android 2.x, of which there is still a non-trivial number of users)
- You have systems that require a wildcard certificate (some unified communications systems do this)
- Anyone who needs a long-lived certificate
- Let’s Encrypt certificates are only valid for 90 days, and must be renewed prior to then. If you need a long-lived certificate, then Let’s Encrypt is not for you
- Anyone who wants Extended Validation
- Let’s Encrypt only validates that you have control over a given domain. It does not validate your identity or business or anything of that nature. As such you cannot get the green security bar that displays in the browser for places like banks or PayPal.
- Anyone who needs their certificate to be trusted by really old things
- If you have devices from 1997 that only trust 1997’s list of CA’s, then you’re going to have a bad time
- However, this is likely the least of your troubles
- Let’s Encrypt is trusted by:
- Android version 2.3.6 and above, released 2011-09-02
- FireFox version 2.0 and above, released 2006-10-24
- Internet Explorer on Windows Vista or above (For Windows XP, see this issue), released 2007-01-30
- Google Chrome on Windows Vista or above (For Windows XP, see this issue), released 2008-08-02
- Safari on OSX v4.0 or above (Mac OSX 10.4 or newer), released 2005-04-29
- Safari on iOS v3.1 or above, released 2010-02-02
However, these are mostly edge cases, and if you’re reading this blog post, then you will know if they apply to you or not.
So let’s get out there and encrypt!
The elephant in the room
“But hang on!”, I hear the eagle-eyed reader say. “Stack Overflow is not using SSL/TLS!” you say. And you would be partly correct.
We do offer SSL on all our main sites. Go ahead, try it:
However, we have some slightly more complicated issues at hand. For details about our issues, see the great blog post by Nick Craver. It’s from 2013 and we have fixed many of the issues that we were facing back then, but there is still some way to go.
However, all our signup and login pages however are delivered over HTTPS, and you can switch to HTTPS manually if you would prefer – for most sites.
Let’s get started
So how do you get started? If you have a debian-based Apache server, then grab the Let’s Encrypt tool and go!
If you’re on a different platform, then check the list of pre-build clients above, or take a look at a recent comparison of the most common *nix scripts.
Addendum: Michael Hampton pointed out to me that Fedora ships with the Let’s Encrypt package as a part of their distribution and is also in EPEL if you’re on RedHat, CentOS or another distribution that can make use of EPEL packages.
Craig Peterson joined us back in February. We neglected to announce him in a timely manner because we’re horrible people. That’s OK though, as he’s now one of us.
In contrast to the rest of the SRE team, Craig’s superpowers come in the form of development. He’s been focused on making our homegrown monitoring system, Bosun, better than all of the monitoring systems that have or will ever exist. As Bosun is open source, you can practically hover over Craig’s shoulder and watch his day to day work. In fact, I encourage you to. I’m sure Craig won’t mind in the least.
Craig hails from the state of Utah, making him our second Utahn (or Utahan) member of the SRE team.
Mark Henderson joins us from the nation of OZ. He has regaled us with tales of drop bears and his strong feelings about the suitability of Foster’s as a consumable liquid.
He has relocated nearly ten-thousand miles away to our New York office, and as such will finally get a slight taste of how December should feel. On the plus side, he will no longer have to contend with a smorgasbord of creatures which actively campaign to wipe out the human race.
Mark has also served as a Server Fault moderator since 2011, and has acquired quite a bit of rep in service of the community.
Please join me in welcoming Craig and Mark to the Site Reliability Engineering team!
A few weeks ago we upgraded a lot of the core infrastructure in our New York (okay, it’s really in New Jersey now – but don’t tell anyone) data center. We love being open with everything we do (including infrastructure), and really consider it one of the best job perks we have. So here’s how and why we upgrade a data center. First, take a moment to look at what Stack Overflow started as. It’s 5 years later and hardware has come a long way.
Up until 2 months ago, we hadn’t replaced any servers since upgrading from the original Stack Overflow web stack. There just hasn’t been a need since we first moved to the New York data center (Oct 23rd, 2010 – over 4 years ago). We’re always reorganizing, tuning, checking allocations, and generally optimizing code and infrastructure wherever we can. We mostly do this for page load performance; the lower CPU and memory usage on the web tier is usually a (welcomed) side-effect.
So what happened? We had a meetup. All of the Stack Exchange engineering staff got together at our Denver office in October last year and we made some decisions. One of those decisions was what to do about infrastructure hardware from a lifecycle and financial standpoint. We decided that from here on out: hardware is good for approximately 4 years. After that we will: retire it, replace it, or make an exception and extend the warranty on it. This lets us simplify a great many things from a management perspective, for example: we limit ourselves to 2 generations of servers at any given time and we aren’t in the warranty renewal business except for exceptions. We can order all hardware up front with the simple goal of 4 years of life and with a 4 year warranty.
Why 4 years? It seems pretty arbitrary. Spoiler alert: it is. We were running on 4 year old hardware at the time and it worked out pretty well so far. Seriously, that’s it: do what works for you. Most companies depreciate hardware across 3 years, making questions like “what do we do with the old servers?” much easier. For those unfamiliar, depreciated hardware effectively means “off the books.” We could re-purpose it outside production, donate it, let employees go nuts, etc. If you haven’t heard, we raised a little money recently. While the final amounts weren’t decided when we were at the company meetup in Denver, we did know that we wanted to make 2015 an investment year and beef up hardware for the next 4.
Over the next 2 months, we evaluated what was over 4 years old and what was getting close. It turns out almost all of our Dell 11th generation hardware (including the web tier) fits these criteria – so it made a lot of sense to replace the entire generation and eliminate a slew of management-specific issues with it. Managing just 12th and 13th generation hardware and software makes life a lot easier – and the 12th generation hardware will be mostly software upgradable to near equivalency to 13th gen around April 2015.
What Got Love
In those 2 months, we realized we were running on a lot of old servers (most of them from May 2010):
- Web Tier (11 servers)
- Redis Servers (2 servers)
- Second SQL Cluster (3 servers – 1 in Oregon)
- File Server
- Utility Server
- VM Servers (5 servers)
- Tag Engine Servers (2 servers)
- SQL Log Database
We also could use some more space, so let’s add on:
- An additional SAN
- An additional DAS for the backup server
I know what you’re thinking: “Nick, how do you go about making such a fancy pile of servers?” I’m glad you asked. Here’s how a Stack Exchange infrastructure upgrade happens in the live data center. We chose not to failover for this upgrade; instead we used multiple points of redundancy in the live data center to upgrade it while all traffic was flowing from there.
Day -3 (Thursday, Jan 22nd): Our upgrade plan was finished (this took about 1.5 days total), including everything we could think of. We had limited time on-site, so to make the best of that we itemized and planned all the upgrades in advance (most of them successfully, read on). You can find a read the full upgrade plan here.
Day 0 (Sunday, Jan 25th): The on-site sysadmins for this upgrade were George Beech, Greg Bray, and Nick Craver (note: several remote sysadmins were heavily involved in this upgrade as well: Geoff Dalgas online from Corvallis, OR, Shane Madden, online from Denver, CO, and Tom Limoncelli who helped a ton with the planning online from New Jersey). Shortly before flying in we got some unsettling news about the weather. We packed our snow gear and headed to New York.
Day 1 (Monday, Jan 26th): While our office is in lower Manhattan, the data center is now located in Jersey City across the Hudson river: We knew there was a lot to get done in the time we had allotted in New York, weather or not. The thought was that if we skipped Monday we likely couldn’t get back to the data center Tuesday if the PATH (mass transit to New Jersey) shut down. This did end up happening. The team decision was: go time. We got overnight gear then headed to the data center. Here’s what was there waiting to be installed:
Yeah, we were pretty excited too. Before we got started with the server upgrade though, we first had to fix a critical issue with the redis servers supporting the launching-in-24-hours Targeted Job Ads. These machines were originally for Cassandra (we broke that data store), then Elasticsearch (broke that too), and eventually redis. Curious? Jason Punyon and Kevin Montrose have an excellent blog series on Providence, you can find Punyon’s post on what broke with each data store here.
The data drives we ordered for these then-redundant systems were the Samsung 840 Pro drives which turned out to have a critical firmware bug. This was causing our server-to-server copies across dual 10Gb network connections to top out around 12MB/s (ouch). Given the hundreds of gigs of memory in these redis instances, that doesn’t really work. So we needed to upgrade the firmware on these drives to restore performance. This needed to be online, letting the RAID 10 arrays rebuild as we went. Since you can’t really upgrade firmware over most USB interfaces, we tore apart this poor, poor little desktop to do our bidding:
Once that was kicked off, it ran in parallel with other work (since RAID 10s with data take tens of minutes to rebuild, even with SSDs). The end result was much improved 100-200MB/s file copies (we’ll see what new bottleneck we’re hitting soon – still lots of tuning to do). Now the fun begins. In Rack C (we have high respect for our racks, they get title casing), we wanted to move from the existing SFP+ 10Gb connectivity combined with 1Gb uplinks for everything else to a single dual 10Gb BASE-T (RJ45 connector) copper solution. This is for a few reasons: The SFP+ cabling we use is called twinaxial which is harder to work with in cable arms, has unpredictable girth when ordered, and can’t easily be gotten natively in the network daughter cards for these Dell servers. The SFP+ FEXes also don’t allow us to connect any 1Gb BASE-T items that we may have (though that doesn’t apply in this rack, it does when making it a standard across all racks like with our load balancers). So here’s what we started with in Rack C:
- 2 Nexus 2232PP 10Gb SFP+ FEXes at the top
- 2 Nexus 2248TP-E 1Gb BASE-T FEXes in the middle
- 1 2960 Management switch
- 2 Avocent KVM aggregator/uplinks
- 6U of Panduit cable management
What we want to end up with is:
- 2 Nexus 2232TM 10Gb BASE-T FEXes
- 1 2960 Management switch
- 1 Avocent KVM aggregator/uplink
- 5U of Panduit cable management
The plan was to simplify network config, cabling, overall variety, and save 4U in the process. Here’s what the top of the rack looked like when we started: …and the middle (cable management covers already off):
Let’s get started. First, we wanted the KVMs online while working so we, ummm, “temporarily relocated” them: Now that those are out of the way, it’s time to drop the existing SFP+ FEXes down as low as we could to install the new 10Gb BASE-T FEXes in their final home up top: The nature of how the Nexus Fabric Extenders work allows us to allocate between 1 and 8 uplinks to each FEX. This means we can unplug 4 ports from each FEX without any network interruption, take the 4 we find dead in the VPC (virtual port channel) out of the VPC and assign them to the new FEX. So we go from 8/0 to 4/4 to 0/8 overall as we move from old to new through the upgrade. Here’s the middle step of that process: With the new network in place, we can start replacing some servers. We yanked several old servers already, one we virtualized and 2 we didn’t need anymore. Combine this with evacuating our NY-VM01 & NY-VM02 hosts and we’ve made 5U of space through the rack. On top of NY-VM01&02 was 1 of the 1Gb FEXes and 1U of cable management. Luckily for us, everything is plugged into both FEXes and we could rip one out early. This means we could spin up the new VM infrastructure faster than we had planned. Yep, we’re already changing THE PLAN™. That’s how it goes. What are we replacing those aging VM servers with? I’m glad you asked. These bad boys:
There are 2 of these Dell PowerEdge FX2s Blade Chassis each with 2 FC630 blades. Each blade has dual Intel E5-2698v3 18-core processors and 768GB of RAM (and that’s only half capacity). Each chassis has 80Gbps of uplink capacity as well via the dual 4x 10Gb IOA modules. Here they are installed:
The split with 2 half-full chassis give us 2 things: capacity to expand by double, and avoiding any single points of failure with the VM hosts. That was easy, right? Well what we didn’t plan on was the network portion of the day, it turns out those IO Aggregators in the back are pretty much full switches with 4 external 10Gbps ports and 8 internal 10Gbps (2 per blade) ports each. Once we figured out what they could and couldn’t do, we got the bonding in place and the new hosts spun up.
It’s important to note here it wasn’t any of the guys in the data center spinning up this VM architecture after the network was live. We’re setup so that Shane Madden was able to do all this remotely. Once he had the new NY-VM01 & 02 online (now blades), we migrated all VMs over to those 2 hosts and were able to rip out the old NY-VM03-05 servers to make more room. As we ripped things out, Shane was able to spin up the last 2 blades and bring our new beasts fully online. The net result of this upgrade was substantially more CPU and memory (from 528GB to 3,072GB overall) as well as network connectivity. The old hosts each had 4x 1Gb (trunk) for most access and 2x 10Gb for iSCSI access to the SAN. The new blade hosts each have 20Gb of trunk access to all networks to split as they need.
But we’re not done yet. Here’s the new EqualLogic PS6210 SAN that went in below (that’s NY-LOGSQL01 further below going in as well):
Our old SAN was a PS6200 with 24x 900GB 10k drives and SFP+ only. This is a newer 10Gb BASE-T 24x 1.2TB 10k version with more speed, more space, and the ability to go active/active with the existing SAN. Along the the SAN we also installed this new NY-LOGSQL01 server (replacing an aging Dell R510 never designed to be a SQL server – it was purchased as a NAS):
The additional space freed by the other VM hosts let us install a new file and utility server:
Of note here: the NY-UTIL02 utility server has a lot of drive bays so we could install 8x Samsung 840 Pros in a RAID 0 in order to restore and test the SQL backups we make every night. It’s RAID 0 for space because all of the data is literally loaded from scratch nightly – there’s nothing to lose. An important lesson we learned last year was that the 840 Pros do not have capacitors in there and power loss will cause data loss if they’re active since they have a bit of DIMM for write cache on board. Given this info – we opted to stick some Intel S3700 800GB drives we had from the production SQL server upgrades into our NY-DEVSQL01 box and move the less resilient 840s to this restore server where it really doesn’t matter.
Okay, let’s snap back to blizzard reality. At this point mass transit had shut down and all hotels in (blizzard) walking distance were booked solid. Though we started checking accommodations as soon as we arrived on site, we had no luck finding any hotels. Though the blizzard did far less than predicted, it was still stout enough to shut everything down. So, we decided to go as late as we could and get ahead of schedule. To be clear: this was the decision of the guys on site, not management. At Stack Exchange employees are trusted to get things done, however they best perceive how to do that. It’s something we really love about this job.
If life hands you lemons, ignore those silly lemons and go install shiny new hardware instead.
This is where we have to give a shout out to our data center QTS. These guys had the office manager help us find any hotel we could, set out extra cots for us to crash on, and even ordered extra pizza and drinks so we didn’t go starving. This was all without asking – they are always fantastic and we’d recommend them to anyone looking for hosting in a heartbeat.
After getting all the VMs spun up, the SAN configured, and some additional wiring ripped out, we ended around 9:30am Tuesday morning when mass transit was spinning back up. To wrap up the long night, this was the near-heart attack we ended on, a machine locking up at: Turns out a power supply was just too awesome and needed replacing. The BIOS did successfully upgrade with the defective power supply removed and we got a replacement in before the week was done. Note: we ordered a new one rather than RMA the old one (which we did later). We keep a spare power supply for each wattage level in the data center, and try to use as few different levels as possible.
Day 2 (Tuesday, Jan 27th): We got some sleep, got some food, and arrived on site around 8pm. Starting the web tier (a rolling build out) was kicked off first:
While we rotated 3 servers at a time out for rebuilds on the new hardware, we also upgraded some existing R620 servers from 4x 1Gb network daughter cards to 2x 10Gb + 2x 1Gb NDCs. Here’s what that looks like for NY-SERVICE03:
The web tier rebuilding gave us a chance to clean up some cabling. Remember those 2 SFP+ FEXes? They’re almost empty: The last 2 items were the old SAN and that aging R510 NAS/SQL server. This is where the first major hiccup in our plan occurred. We planned to install a 3rd PCIe card in the backup server pictured here: We knew it was a Dell R620 10 bay chassis that has 3 half-height PCIe cards. We knew it had a SAS controller for the existing DAS and a PCIe card for the SFP+ 10Gb connections it has (it’s in the network rack with the cores in which all 96 ports are 10Gb SFP+). Oh hey look at that, it’s hooked to a tape drive which required another SAS controller we forgot about. Crap. Okay, these things happen. New plan.
We had extra 10Gb network daughter cards (NDCs) on hand, so we decided to upgrade the NDC in the backup server, remove the SFP+ PCIe card, and replace it with the new 12Gb SAS controller. We also forgot to bring the half-height mounting bracket for the new card and had to get creative with some metal snips (edit: turns out it never came with one – we feel slightly less dumb about this now). So how do we plug that new 10Gb BASE-T card into the network core? We can’t. At least not at 10Gb. Those 2 last SFP+ items in Rack C also need a home – so we decided to make a trade. The whole backup setup (including new MD1400 DAS) just love their new Rack C home:
Then we could finally remove those SFP+ FEXes, bring those KVMs back to sanity, and clean things up in Rack C:
See? There was a plan all along. The last item to go in Rack C for the day is NY-GIT02, our new Gitlab and TeamCity server:
Note: we used to run TeamCity on Windows on NY-WEB11. Geoff Dalgas threw out the idea during the upgrade of moving it to hardware: the NY-GIT02 box. Because they are such intertwined dependencies (for which both have an offsite backup), combining them actually made sense. It gave TeamCity more power, even faster disk access (it does a lot of XML file…stuff), and made the web tier more homogenous all at the same time. It also made the downtime of NY-WEB11 (which was imminent) have far less impact. This made lots of sense, so we changed THE PLAN™ and went with it. More specifically, Dalgas went with it and set it all up, remotely from Oregon. While this is happening, Greg was fighting with a DSC install hang regarding git on our web tier: Wow that’s a lot of red, I wonder who’s winning. And that’s Dalgas in a hangout on my laptop, hi Dalgas! Since the web tier builds were a relatively new process fighting us, we took the time to address some of the recent cabling changes. The KVMs were installed hastily not long before this because we knew a re-cable was coming. In Rack A for example we moved the top 10Gb FEX up a U to expand the cable management to 2U and added 1U of management space between the KVMs. Here’s that process:
Since we had to re-cable from the 1Gb middle FEXes in Rack A & B (all 4 being removed) to the 10Gb Top-of-Rack FEXes, we moved a few things around. The CloudFlare load balancers down below the web tier at the bottom moved up to spots freed by the recently virtualized DNS servers to join the other 2 public load balancers. The removal of the 1Gb FEXes as part of our all-10Gb overhaul meant that the middle of Racks A & B had much more space available, here’s the before and after:
After 2 batches of web servers, cable cleanup, and network gear removal, we called it quits around 8:30am to go grab some rest. Things were moving well and we only had half the web tier, cabling, and a few other servers left to replace.
Day 3 (Wednesday, Jan 28th): We were back in the data center just before 5pm, set up and ready to go. The last non-web servers to be replaced were the redis and “service” (tag engine, elasticsearch indexing, etc.) boxes:
We have 3 tag engine boxes (purely for reload stalls and optimal concurrency, not load) and 2 redis servers in the New York data center. One of the tag engine boxes was a more-recent R620, (this one got the 10Gb upgrade earlier) and wasn’t replaced. That left NY-SERVICE04, NY-SERVICE05, NY-REDIS01 and NY-REDIS02. On the service boxes the process was pretty easy, though we did learn something interesting: if you put both of the drives from the RAID 10 OS array in an R610 into the new R630…it boots all the way into Windows 2012 without any issues. This threw us for a moment because we didn’t remember building it in the last 3 minutes. Rebuild is simple: lay down Windows 2012 R2 via our image + updates + DSC, then install the jobs they do. StackServer (from a sysadmin standpoint) is simply a windows service – and our TeamCity build handles the install and such, it’s literally just a parameter flag. These boxes also run a small IIS instance for internal services but that’s also a simple build out. The last task they do is host a DFS share, which we wanted to trim down and simplify the topology of, so we left them disabled as DFS targets and tackled that the following week – we had NY-SERVICE03 in rotation for the shares and could do such work entirely remotely. For redis we always have a slave chain happening, it looks like this: This means we can do an upgrade/failover/upgrade without interrupting service at all. After all those buildouts, here’s the super fancy new web tier installed:
To get an idea of the scale of hardware difference, the old web tier was Dell R610s with dual Intel E5640 processors and 48GB of RAM (upgraded over the years). The new web tier has dual Intel 2687W v3 processors and 64GB of DDR4 memory. We re-used the same dual Intel 320 300GB SSDs for the OS RAID 1. If you’re curious about specs on all this hardware – the next post we’ll do is a detailed writeup of our current infrastructure including exact specs.
Day 4 (Thursday, Jan 29th): I picked a fight with the cluster rack, D. Much of the day was spent giving the cluster rack a makeover now that we had most of the cables we needed in. When it was first racked, the pieces we needed hadn’t arrived by go time. It turns out we were still short a few cat and power cables as you’ll see in the photos, but we were able to get 98% of the way there.
It took a while to whip this rack into shape because we added cable arms where they were missing, replaced most of the cabling, and are fairly particular about the way we do things. For instance: how do you know things are plugged into the right port and where the other end of the cable goes? Labels. Lots and lots of labels. We label both ends of every cable and every server on both sides. It adds a bit of time now, but it saves both time and mistakes later.
Here’s what the racks ended up looking like when we ran out of time this trip:
It’s not perfect since we ran out of several cables of the proper color and length. We have ordered those and George will be tidying the last few bits up.
I know what you’re thinking. We don’t think that’s enough server eye-candy either.
What Went Wrong
- We’d be downright lying to say everything went smoothly. Hardware upgrades of this magnitude never do. Expect it. Plan for it. Allow time for it.
- Remember when we upgraded to those new database servers in 2010 and the performance wasn’t what we expected? Yeah, that. There is a bug we’re currently helping Dell track down in their 1.0.4/1.1.4 BIOS for these systems that seems to not respect whatever performance setting you have. With Windows, a custom performance profile disabling C-States to stay at max performance works. In CentOS 7, it does not – but disabling the Intel PState driver does. We have even ordered and just racked a minimal R630 to test and debug issues like this as well as test our deployment from bare metal to constantly improve our build automation. Whatever is at fault with these settings not being respected, our goal is to get that vendor to release an update addressing the issue so that others don’t get the same nasty surprise.
- We ran into an issue deploying our web tier with DSC getting locked up on a certain reboot thinking it needed a reboot to finish but coming up in the same state after a reboot in an endless cycle. We also hit issues with our deployment of the git client on those machines.
- We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.
- We learned that if you move the drives from a RAID array from an R610 to an R630 and don’t catch the PXE boot prompt, the server will happily boot all the way into the OS.
- We learned the good and the bad of the Dell FX2 IOA architecture and how they are self-contained switches.
- We learned the CMC (management) ports on the FX2 chassis are effectively a switch. We knew they were suitable for daisy chaining purposes. However, we promptly forgot this, plugged them both in for redundancy and created a switching loop that reset Spanning Tree on our management network. Oops.
- We learned the one guy on twitter who was OCD about the one upside down box was right. It was a pain to flip that web server over after opening it upside down and removing some critical box supports.
- We didn’t mention this was a charge-only cable. Wow, that one riled twitter up. We appreciate the #infosec concern though!
- We drastically underestimated how much twitter loves naked servers. It’s okay, we do too.
- We learned that Dell MD1400 (13g and 12Gb/s) DAS (direct attached storage) arrays do not support hooking into their 12g servers like our R620 backup server. We’re working with them on resolving this issue.
- We learned Dell hardware diagnostics don’t even check the power supply, even when the server has an orange light on the front complaining about it.
- We learned that Blizzards are cold, the wind is colder, and sleep is optional.
Here’s what the average render time for question pages looks like, if you look really closely you can guess when the upgrade happened: The decrease on question render times (from approx 30-35ms to 10-15ms) is only part of the fun. The next post in this series will detail many of the other drastic performance increases we’ve seen as the result of our upgrades. Stay tuned for a lot of real world payoffs we’ll share in the coming weeks.
Does all this sound like fun?
To us, it is fun. If you feel the same way, come do it with us. We are specifically looking for sysadmins preferably with data center experience to come help out in New York. We are currently hiring 2 positions:
If you’re curious at all, please ask us questions here, Twitter, or wherever you’re most comfortable. Really. We love Q&A.
Jason hails from Alaska and although he cannot see Russia from his house, he is officially our most Northwestern US Employee beating Geoff Dalgas by over 1500 miles. He will be working remotely except when we decide to fly him to the mainland to help move some servers or sample the excellent work of the NYC Stack Exchange Chefs.
Jason brings a wealth of knowledge to our team from his previous roles working at reddit and Rackspace, and he has already been extremely helpful in our ongoing implementation of CloudFlare. He also has a deep understanding of the complex issues that surround our industry and shares our belief of being open and transparent whenever possible.
We believe Jason is an excellent addition to our SRE team so I hope you will join me in welcoming him!
Imagine if alerting was what you wanted it to be:
- Every alert you received was actionable, and there were few false alerts
- Notifications were actually informative
- You received alerts in time to fix problems before they impacted your users
This isn’t the world we live in…
- We accept lots of notifications from our alerting system that are not actionable
- The notifications don’t tell us about the problem
- We get paged when stuff is dead and not when it is sick
In order to resolve the dissonance between reality and what alerting should be we need:
- An expressive way to evaluate alert conditions that isn’t a 1:1 mapping to the metrics
- Alerts backed by time-series and not just recent values
- A way to to make rich notifications that include useful information
- A way to iterate fast with alert design so that our alerts are continuously improved
A little less than a year ago, Matt Jibson and Kyle Brandt set out to create a system to solve this and other problems in monitoring; we call it Bosun. Our belief is that achieving excellence in alerting is a complex problem and requires a powerful and flexible platform to design alerts. Therefore, Bosun’s strategy is to provide a framework that enables the operator to create intelligent and informative alerts. We believe that you are smarter and more creative than any monitoring system can be when it comes to your environment.
In order to achieve that, at the highest level Bosun provides:
- An expression language (a small domain-specific language) designed to allow for the creation of highly flexible and specific alerts
- Notification templates that allow you to include whatever information you think is relevant
- A web interface the provides a workflow for more rapid iteration with improving and creating alerts: Graph -> Expression -> Rule + Template -> Test the rule over history
The Expression Language
We believe that every alert requires action. An alert asks for your attention, and human attention and time is a valuable asset. So alerting is about owning the operators attention. Taking action with alerts practically means one of two things. If the alert was accurate, then you fix the issue that triggered the alert. If the alert was a false positive, then the alert should be tuned in a way that the false positive won’t trigger the alert. This is where things tend to fall down because alert evaluations are not powerful enough to be tuned. With Bosun’s expression language, you can tune alerts in the following ways:
- Alert thresholds based on history vs static thresholds (or both combined)
- Statistics functions: Min, Percentile, Median, Deviations, Forecasting. You can change the duration that these evaluate over (i.e. 5 minutes, 1 hour, 1 week?)
- Scope-aware: How should components in your environment be grouped? By Host, subsystem, cluster, a combination of those things
- Boolean conditions: The interaction of multiple components
These possibilities, when applied selectively by a skilled operator, provide ample ways to reduce alerting noise.
Once you have someone’s attention with a valid alert, you need to direct them to the problem as accurately as possible. Our notification templates use the Go template language, which means they can be quite flexible. Notifications in Bosun allow you to:
- Include breakdowns of information related to your alert as embedded graphs, html tables, or whatever else you think makes sense
- Include information that wasn’t directly related to the alert: i.e. CPU of a host even though it was a memory alert
- Generate links to your dashboards or other sources of information
- Includes notes about why you created that alert, caveats, and other information the person being notified should be aware of
One of the main issues with alerting is that there is so much friction to tuning alerts that it doesn’t get done. One of Bosun’s goals was to provide a faster iteration cycle for creating and tuning alerts by making the web interface an alerting IDE: Graphs in Bosun’s interface link to expressions, which then link to alert rules and templates. You can then test alerts before implementing; the results of a rule and template can be tested in the interface. You can test how they will behave currently, how they might have behaved at a past time, or generate a timeline of how they might have behaved over a range of time.
This means that your alert tuning doesn’t need to be totally reactionary. You can test alert changes and see how and when they would have triggered over the past weeks (or longer, if you are patient). This results in less alert noise being sent to operators.
But wait! There’s more!
Bosun has also attempted to make some problems in monitoring easier:
- Getting data into the system: our agent (called “scollector”) runs on Windows and Linux and starts sending data to Bosun
- Applications can push metrics to the system via JSON API calls
- Human maintenance: Properly designed alerts will apply to new systems, and services are auto-discovered by scollector. This means you don’t have to remember to update your monitoring most of the time when a new services and hosts are deployed (as long as scollector is pushed out via your build or configuration management process)
We hope you go try this out. We have a docker image that has everything you need—just follow the getting started guide. We hope Bosun is useful to the community. We need your creativity and ideas to continue to grow it (and some contributors would be nice too!). We owe a special thanks to everyone else at Stack Exchange for:
- Contributing to scollector – Greg Bray has been working hard to fill out our Windows metrics, and Sam Torno did the same for Linux
- Getting a docker build – Peter Grace (who also did a lot of the dogfooding)
- Manning the front lines to keep the site up while we built this – the rest of the SRE Team
- Feature ideas and monitoring concepts – Tom Limoncelli and his monitoring chapters in The Practice of Cloud System Administration
- Letting Matt and I go tilting at windmills – Stack Exchange, Inc.
Greg joins us as our new Windows-focused generalist (in case you missed it, our friend Steven Murawski moved on recently), though he’s happy to work on whatever technology we throw at him. He’s a software developer turned sysadmin, and he’ll be assisting us in our quest to automate our infrastructure until it achieves sentience.
Greg is a participant on a number of the sites in the Stack Exchange network (since the early days in 2008!), a University of Utah Computer Engineering graduate (where he worked on a FPGA-based tester for NAND flash storage [pdf link]), and an occasional blogger. He has a knack for finding obscure bugs.
Greg lives in Salt Lake City with his wife and 4 year old cat named Kitty, and he’ll be working remotely from there. When he’s not working with technology, he enjoys biking, camping, and golfing.
Join me in welcoming Greg to our team!
The ServerFault Systems Administration team continues its growth with the addition of sysadmin Shane Madden. Shane lives in Denver and will work out of the Denver office on days he wants lunch.
You may know Shane already from his contributions in the open source world. He’s an avid Puppet programmer and Python coder. Shane’s very active on Server Fault… he has over 69k reputation points which makes him the 7th highest rep user there!
Shane’s hobbies include skiing, board games, video games and hockey. He has many pets including a mantis shrimp.
Please join me in welcoming Shane to the team!
Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.
How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.
Observability is the Foundation
Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:
- Questions operators have about their systems can be quantifiably answered with minimal effort
- Operators have rich mental models of how their systems function
When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.
Decision Making and Incident Preparedness
Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.
OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.
Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.
In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.
You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.
Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.
Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.
Convinced? 5 Steps to Achieving Good Observability:
In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.
- Instrument your systems by publishing metrics and events
- Gather those metrics and events in a queryable data store(s)
- Make that data readily accessible
- Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
- Establish the resources to drill down into abnormal or out of bounds behavior
Each of these steps largely depends on the previous step to be successful.
1. Instrument your Systems
Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:
- Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
- Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.
Good Metrics also tend to have these properties:
- High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
- Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
- Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.
It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.
2. Gather those metrics in a queryable data store(s)
This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:
- Building dashboards
- Enabling capacity planning
- Allowing operators to explore the data and learn
- Allowing people to invent cool stuff you didn’t anticipate
As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:
- Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
- Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views
3. Make that data Readily Accessible
If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:
- A fast responsive UI to allow for operators to drill down and explore easily
- Enables operators to create their own dashboards and graphs
- Highlight problems
4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.
Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.
Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.
5. Establish the resources to drill down into abnormal or out of bounds behavior
The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:
- Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
- Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
- Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)
Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.
Use the science, Luke
If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.
A lot of tools available in IT/Sysadmin/Ops/DevOps are disappointing:
- They don’t fit your environment. They lack features or our designed for a different sort of environment (i.e cloud vs hardware, Linux vs Windows, distributed vs centralized etc)
- You can’t interact with them programmatically
- They cost too much
- They are not customizable enough, or require too much customization to get off the ground
- Feel kludgy, unreliable, outdated, or like the programmers were stoned
- Don’t fit with your company’s culture (i.e. Enterprise vs Agile)
In short a lot of stuff is too expensive, isn’t a good fit, or is simply bad software. This ends up leaving an ops team with two options. They can whine about it, or create their own tools. So at Stack Exchange we build our own DevOps tools.
Nick Craver’s baby, which we just call “Status” is at first glance a monitoring dashboard, but is essentially a collection of tools that filled various needs:
- An Overview of CPU, Memory, and Network utilization for all our servers as well as a detailed view. Done with responsive and interactive D3 graphs as well as sparklines it helps compensate for Solar Wind’s terrible interface.
- SQL Server monitoring. SQL’s built in Clustering views are deeply flawed. If a node loses connectivity, it stops updating remote nodes status, so it could show everything as connected and fine, even if there is no connectivity. We also get to see the most expensive queries, active queries utilizing whoisactive, current connections, and which DBs are on which server
- HAProxy Monitoring and Administration: With multiple instances of HAProxy we needed a single view instead of HAProxy’s built-in display. Also, this gave us a nice web interface to take servers out of rotation
- Redis: A nice presentation of Redis Info across all instances and all servers. Also a display that shows what is slaved to what in at a quick glance
- Elastic Search: Health overview of or clusters (as well as index and shard data)
- A dashboard of all the exceptions generated by our applications
Status is C# / .NET app. It polls data from various sources – sometimes the system directly and other times it gets it from Orion. There is a lot more to status that makes it awesome. The real accomplishment is that status enables us to see the general health of our main infrastructure at a glance.
If you business is creating and running websites, your web logs are gold. We use the logs generated by our load balancer, HAProxy, as our canonical web logs. In their raw text format, web logs are often not that useful (this is particularly true with over 100 million records a day). However we parse and structure our web logs in a few different ways:
- We have C# service that Jarrod Dixon wrote that inserts them into SQL so we can query them. In order to query them we use an instance of Data Explorer, SQL management studio, and also have certain lookups directly from our sites
- Displaying realtime graphs of various log information with Realog, a system I created with Go, Redis, and NVD3.js so we could view activity live without having to write queries
One of the interesting things we do with our weblogs is to add extra information by adding headers inside the app and striping them from the response at HAProxy. For example, we capture how many Redis and SQL queries were involved in that request and how long they took.
- View the outstanding patches and patch count for both Linux and Windows
- Trigger updates on either Linux or Windows
- Schedule time frames for automatic Linux updates
If you want to learn more about these tools and DevOps at Stack Exchange, come see George, Nick, and Steven present “Building for Operations” at Velocity.
Keeping all this stuff to ourselves feels a bit greedy. However, for something open sourced to be very useful it usually needs to be made a bit more generic which takes time. We also want to build a lot more. Our inventory system Racktables lacks an API so we need a new one or a way to extend it. We want to build our own monitoring system (likely on top of OpenTSDB). In order to create more, and open source it we need help. So we are looking a full time developer with ops experience to join our SRE team. So if you are awesome, want to build awesome ops stuff and open source it, come join us!
I’m not talking about data center cooling here…
I was recently listening to The Ship Show podcast titled “Keep calm and PROD on”. In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.
The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents. The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.
The hosts outlined a couple of scenarios for discussion. They didn’t use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..). In no particular order (going from memory and making up the classifications here), we have the “cowboy” response, the “neophyte” response, and the “deer-in-the-headlights” response. I’ll provide a quick synopsis of these responses.
The “Cowboy” Response
This incident response archetype is very identifiable in many environments. Picture it, the year is 2010, it’s 2 AM and your cell phone is blowing up with monitoring alerts. The website is down and everyone is waking up. You log in to your VPN and start gathering information. Everything goes dark, and then everything comes back up, seemingly magically fixed. It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.
This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I’ve observed over my years in IT.
The “Neophyte” Response
This incident response archetype is your typical “newbie”. The neophyte does not have to be new to the field, just new to high pressure outage scenarios. Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up. The neophyte might not be comfortable with getting a more senior engineer out of bed, or getting some developers involved. Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.
The “Deer-In-The-Headlights” Response
The last one I remember the podcast covering was the “deer-in-the-headlights” response, where regardless of a person’s experience level, they just don’t respond well in a crisis. Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders. This leads to a fractured response, where people may be working at cross purposes.
What Should We Do?
What I’ve Learned In Previous Careers
Training To Be A Cop
You may not know this about me, but before I got into IT, I was training to be a cop. I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns. A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations. (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)
In that training, we drilled how to respond to emergencies as individuals and as part of a team. Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes. Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions. Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.
Part of the training regimen includes working past the failures and mistakes. In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we’ve drilled in the classroom and on the range. We are coached to get to cover, return fire, and call for backup. We can’t dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.
Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits. In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for. We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera. This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills. Training helps minimize the “deer-in-the-headlights” and the “neophyte” responses and allows people who would experience those responses to fall back on training.
Working In A Public Safety Agency
I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department. Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations. I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.
Under the NIMS model, the first responding public safety officer is the incident commander. As the situation develops, the incident commander role can change, based on who’s best suited to deal with the incident. For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command. (In my experience, this is after police officers have saved all the people in the building. 😉 ). Let’s look at another scenario: officers respond to a report of a burglary in progress. The first officer responding is the incident commander. As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation. Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.
NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer. The PIO’s responsibility is to keep the stakeholders and public in general informed as to what is going on. Part of the role is determining what information is helpful to share and what should not be disclosed. The Safety Officer monitors conditions and ensures the safety of all incident personnel. The Liaison Officer is responsible for dealing with all the coordinating agencies. Defining these roles can help deal with the “cowboy” response. If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus “just get x involved, he fixed it last time.”
How We Should Respond
These two experiences provide some basic thoughts into how we can approach incident response.
Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn’t immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.
For operations personnel (developers, dba’s – yes them too – and sysadmins), this is critical.
For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel. Guess what? The knowledgeable guy isn’t always around when things go ill.
Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital. Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally. Quick – what do you do when your CDN stops serving content? Have you drilled that scenario? If you haven’t where do you even begin?
This is something we are going to be focusing more on here at Stack Exchange and I’m super excited about that. We’ve decided that this is a priority for our organization and we’ll be dedicating time to this.
In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations. The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team. Certain situations we can just deal with and not need to involve others. If I need backup (additional resources), I can request those, but if a situation escalates, it’s time to bring in more support. We don’t have a strict command and control environment; as we grow our technical staff I think that’ll be more defined.
When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience. Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role. We currently don’t have anything defined like that, though I hope as we start to drill these scenarios more, I’ll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.
In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be. By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.
In our case, I can envision that as having our PIO designate one chat room or google hangout as our internal status update location and during the incident he/she’d make regular updates to those areas. In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.
As for the Incident Commander role, we’d need to train all of our on-call personnel, as well the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident. While we don’t have this defined yet, we had a short outage a few weeks back that illustrates how this can work.
- About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom. I wasn’t the on-call person, but I always monitor for severe external alert failures.
- I rolled out of bed and
ranstumbled down to my office and got online to start investigating.
- First order of business, check our chat site.. oops! It’s down! Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it’s running in New York with the rest of our infrastructure.
- Next up, VPN.. which connects.. that means internet to our data center in NY is still good.
- I jump in a Google Hangout (
fortunatelyunfortunately for my co-workers, I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team.
- Chat comes back online. I dropped a note in chat that I’m in the SRE hangout and troubleshooting.
- I’m soon joined by Geoff Dalgas, one of our Core Q&A developers.
- He and I discuss the situation and began validating the different bit of our infrastructure.
- We determined that we were seeing an issue with keepalived on our load balancers.
- Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
- We also had several more developers pop in and see if they could offer any help.
- We determined a course of action to remediate the problem and began to implement it.
- Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.
In this situation, I was first on-scene and acted as incident commander. If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.
So, what do you do in your organization? Do you drill and train for failures? Do you test your backups? Do you prepare your operations personnel for how to respond in an incident?