DNS at Stack Overflow is… complex. We have hundreds of DNS domains and thousands of DNS records. We have gone from running our own BIND server to hosting DNS with multiple cloud providers, and we change things fairly often. Keeping everything up to date and synced at multiple DNS providers is difficult. We built DnsControl to allow us to perform updates easily and automatically across all providers we use.

The old way

Originally, our DNS was hosted by our own BIND servers, using artisanal, hand crafted zone files. Large changes involved liberal sed usage, and every change was pretty error prone. We decided to start using cloud DNS providers for performance reasons, but those each have their own web panels, which are universally painful to use. Web interfaces rarely have any import/export functionality, and generally lack change control, history tracking, or comments. We quickly decided that web panels were not how we wanted to manage our zones.

Introducing DnsControl

DNSControl is the system we built to manage our DNS. It permits “describe once, use anywhere” DNS management. It consists of a few key components:

1. A Domain Specific Language (DSL) for describing domains in a single, provider-independent way.
2. An “interpreter” application that executes the DSL and creates a standardized representation of your desired DNS state.
3. Back-end “providers” that sync the desired state to a DNS provider.

At the time of this writing we have 9 different providers implemented, with 3 more on the way shortly. We use it to manage our domains with our own BIND servers, as well as Route 53, Google Cloud DNS, name.com, Cloudflare, and more.

A sample might look like this description of stackoverflow.com:

D(“stackoverflow.com”, REG_NAMEDOTCOM, DnsProvider(R53), DnsProvider(GCLOUD),
    A(“@”, “198.252.206.16”),
    A(“blog”, “198.252.206.20”),
    CNAME(“chat”, “chat.stackexchange.com.”),
    CNAME(“www”, “@”, TTL(3600)),
    A(“meta”, “198.252.206.16”)
)

This is just a small, simple example. The DSL is a fully-featured way to express your DNS config. It is actually just javascript with some helpful functions. We have an examples page with more examples of the power of the language.

Running “dnscontrol preview” with this input will show what updates would be needed to bring DNS providers up to date with the new, desired, configuration. “dnscontrol push” will actually make the changes.

This allows us to manage our DNS configuration as code. Storing it this way has a bunch of advantages:

• We can use variables to store common IP addresses or repeated data. We can make complicated changes, like failing-over services between data centers, by changing a single variable. We can activate or deactivate our CDN, which involves thousands of record changes, by commenting or uncommenting a single line of code.
• We are not locked into any single provider, since the automation can sync to any of them. Keeping records synchronized between different cloud providers requires no manual steps.
• We store our DNS config in git. Our build server runs all changes. We have central logging, access control, and history for our DNS changes. We’re trying to apply DevOps best practices to an area that has not seen those benefits so much yet.

I think the biggest benefit to this tool though is the freedom it has given us with our DNS. It has allowed us to:

• Switch providers with no fear of breaking things. We have changed CDNs or DNS providers at least 4 times in the last two years, and it has never been scary at all.
• Dual-host our DNS with multiple providers simultaneously. The tool keeps them in sync for us.
• Test fail-over procedures before an emergency happens. We are confident we can point DNS at our secondary datacenter easily, and we can quickly switch providers if one is being DDOSed.

DNS configuration is often difficult and error-prone. We hope DnsControl makes it easy and more reliable. It has for us.

Some resources:

• The github repo
• The documentation site (with Getting Started Guide)
• A talk I gave at SRECon 2017 Americas

Let’s talk about DNS. After all, what could go wrong? It’s just cache invalidation and naming things .

tl;dr

This blog post is about how Stack Overflow and the rest of the Stack Exchange network approaches DNS:

• By bench-marking different DNS providers and how we chose between them
• By implementing multiple DNS providers
• By deliberately breaking DNS to measure its impact
• By validating our assumptions and testing implementations of the DNS standard

The good stuff in this post is in the middle, so feel free to scroll down to “The Dyn Attack” if you want to get straight into the meat and potatoes of this blog post.

The Domain Name System

DNS had its moment in the spotlight in October 2016, with a major Distributed Denial of Service (DDos) attack launched against Dyn , which affected the ability for Internet users to connect to some of their favourite websites, such as Twitter, CNN, imgur, Spotify, and literally thousands of other sites.

But for most systems administrators or website operators, DNS is mostly kept in a little black box, outsourced to a 3rd party, and mostly forgotten about. And, for the most part, this is the way it should be. But as you start to grow to 1.3+ billion pageviews a month with a website where performance is a feature, every little bit matters.

In this post, I’m going to explain some of the decisions we’ve made around DNS in the past, and where we’re going with it in the future. I will eschew deep technical details and gloss over low-level DNS implementation in favour of the broad strokes.

In the beginning

So first, a bit of history: In the beginning, we ran our own DNS on-premises using artisanally crafted zone files with BIND. It was fast enough when we were doing only a few hundred million hits a month, but eventually hand-crafted zonefiles were too much hassle to maintain reliably. When we moved to Cloudflare as our CDN, their service is intimately coupled with DNS, so we demoted our BIND boxes out of production and handed off DNS to Cloudflare.

The search for a new provider

Fast forward to early 2016 and we moved our CDN to Fastly. Fastly doesn’t provide DNS service, so we were back on our own in that regards and our search for a new DNS provider began. We made a list of every DNS provider we could think of, and ended up with a shortlist of 10:

• Dyn
• NS1
• Amazon Route 53
• Google Cloud DNS
• Azure DNS (beta)
• DNSimple
• Godaddy
• EdgeCast (Verizon)
• Hurricane Electric
• DNS Made Easy

From this list of 10 providers, we did our initial investigations into their service offerings, and started eliminating services that were either not suited to our needs, outrageously expensive, had insufficient SLAs, or didn’t offer services that we required (such as a fully featured API). Then we started performance testing. We did this by embedding a hidden iFrame on 5% of the visitors to stackoverflow.com, which forced a request to a different DNS provider. We did this for each provider until we had some pretty solid performance numbers.

Using some basic analytics, we were able to measure the real-world performance, as seen by our real-world users, broken down into geographical area. We built some box plots based on these tests which allowed us to visualise the different impact each provider had.

If you don’t know how to interpret a boxplot, here’s a brief primer for you. For the data nerds, these were generated with R’s standard boxplot functions, which means the upper and lower whiskers are min(max(x), Q_3 + 1.5 * IQR) and max(min(x), Q_1 – 1.5 * IQR), where IQR = Q_3 – Q_1

This is the results of our tests as seen by our users in the United States:

You can see that Hurricane Electric had a quarter of requests return in < 16ms and a median of 32ms, with the three “cloud” providers (Azure, Google Cloud DNS and Route 53) being slightly slower (being around 24ms first quarter and 45ms median), and DNS Made Easy coming in 2nd place (20ms first quarter, 39ms median).

You might wonder why the scale on that chart goes all the way to 700ms when the whiskers go nowhere near that high. This is because we have a worldwide audience, so just looking at data from the United States is not sufficient. If we look at data from New Zealand, we see a very different story:

Here you can see that Route 53, DNS Made Easy and Azure all have healthy 1st quarters, but Hurricane Electric and Google have very poor 1st quarters. Try to remember this, as this becomes important later on.

We also have Stack Overflow in Portuguese, so let’s check the performance from Brazil:

Here we can see Hurricane Electric, Route 53 and Azure being favoured, with Google and DNS Made Easy being slower.

So how do you reach a decision about which DNS provider to choose, when your main goal is performance? It’s difficult, because regardless of which provider you end up with, you are going to be choosing a provider that is sub-optimal for part of your audience.

You know what would be awesome? If we could have two DNS providers, each one servicing the areas that they do best! Thankfully this is something that is possible to implement with DNS. However, time was short, so we had to put our dual-provider design on the back-burner and just go with a single provider for the time being.

Our initial rollout of DNS was using Amazon Route 53 as our provider: they had acceptable performance figures over a large number of regions and had very effective pricing (on that note Route 53, Azure DNS, and Google Cloud DNS are all priced identically for basic DNS services).

The DYN attack

Roll forwards to October 2016. Route 53 had proven to be a stable, fast, and cost-effective DNS provider. We still had dual DNS providers on our backlog of projects, but like a lot of good ideas it got put on the back-burner until we had more time. Then the Internet ground to a halt. The DNS provider Dyn had come under attack, knocking a large number of authoritative DNS servers off the Internet, and causing widespread issues with connecting to major websites. All of a sudden DNS had our attention again. Stack Overflow and Stack Exchange were not affected by the Dyn outage, but this was pure luck.

We knew if a DDoS of this scale happened to our DNS provider, the solution would be to have two completely separate DNS providers. That way, if one provider gets knocked off the Internet, we still have a fully functioning second provider who can pick up the slack. But there were still questions to be answered and assumptions to be validated:

• What is the performance impact for our users in having multiple DNS providers, when both providers are working properly?
• What is the performance impact for our users if one of the providers is offline?
• What is the best number of nameservers to be using?
• How are we going to keep our DNS providers in sync?

These were pretty serious questions – some of which we had hypothesis that needed to be checked and others that were answered in the DNS standards, but we know from experience that DNS providers in the wild do not always obey the DNS standards.

What is the performance impact for our users in having multiple DNS providers, when both providers are working properly?

This one should be fairly easy to test. We’ve already done it once, so let’s just do it again. We fired up our tests, as we did in early 2016, but this time we specified two DNS providers:

• Route 53 & Google Cloud
• Route 53 & Azure DNS
• Route 53 & Our internal DNS

We did this simply by listing Name Servers from both providers in our domain registration (and obviously we set up the same records in the zones for both providers).

Running with Route 53 and Google or Azure was fairly common sense – Google and Azure had good coverage of the regions that Route 53 performed poorly in. Their pricing is identical to Route 53, which would make forecasting for the budget easy. As a third option, we decided to see what would happen if we took our formerly demoted, on-premises BIND servers and put them back into production as one of the providers. Let’s look at the data for the three regions from before: United States, New Zealand and Brazil:

United States

New Zealand

Brazil

There is probably one thing you’ll notice immediately from these boxplots, but there’s also another not-so obvious change:

1. Azure is not in there (the obvious one)
2. Our 3rd quarters are measurably slower (the not-so obvious one).

Azure

Azure has a fatal flaw in their DNS offering, as of the writing of this blog post. They do not permit the modification of the NS records in the apex of your zone:

You cannot add to, remove, or modify the records in the automatically created NS record set at the zone apex (name = “@”). The only change that’s permitted is to modify the record set TTL.

These NS records are what your DNS provider says are authoritative DNS servers for a given domain. It’s very important that they are accurate and correct, because they will be cached by clients and DNS resolvers and are more authoritative than the records provided by your registrar.

Without going too much into the actual specifics of how DNS caching and NS records work (it would take me another 2,500 words to describe this in detail), what would happen is this: Whichever DNS provider you contact first would be the only DNS provider you could contact for that domain until your DNS cache expires. If Azure is contacted first, then only Azure’s nameservers will be cached and used. This defeats the purpose of having multiple DNS providers, as in the event that the provider you’ve landed on goes offline, which is roughly 50:50, you will have no other DNS provider to fall back to.

So until Azure adds the ability to modify the NS records in the apex of a zone , they’re off the table for a dual-provider setup.

The 3rd quarter

What the third quarter represents here is the impact of latency on DNS. You’ll notice that in the results for ExDNS (which is the internal name for our on-premises BIND servers) the box plot is much taller than the others. This is because those servers are located in New Jersey and Colorado – far, far away from where most of our visitors come from. So as expected, a service with only two points of presence in a single country (as opposed to dozens worldwide) performs very poorly for a lot of users.

Performance conclusions

So our choices were narrowed for us to Route 53 and Google Cloud, thanks to Azure’s lack of ability to modify critical NS records. Thankfully, we have the data to back up the fact that Route 53 combined with Google is a very acceptable combination.

Remember earlier, when I said that the performance of New Zealand was important? This is because Route 53 performed well, but Google Cloud performed poorly in that region. But look at the chart again. Don’t scroll up, I’ll show you another chart here:

See how Google on its own performed very poorly in NZ (its 1st quarter is 164ms versus 27ms for Route 53)? However, when you combine Google and Route 53 together, the performance basically stays the same as when there was just Route 53.

Why is this? Well, it’s due to a technique called Smooth Round Trip Time. Basically, DNS resolvers (namely certain version of BIND and PowerDNS) keep track of which DNS servers respond faster, and weight queries towards those DNS servers. This means that the faster provider should be skewed to more often than the slower providers. There’s a nice presentation over here if you want to learn more about this. The short version is that if you have many DNS servers, DNS cache servers will favour the fastests ones. As a result, if one provider is fast in Auckland but slow in London, and another provider is the reverse, DNS cache servers in Auckland will favour the first provider and DNS cache servers in London will favor the other. This is a very little known feature of modern DNS servers but our testing shows that enough ISPs support it that we are confident we can rely on it.

What is the performance impact for our users if one of the providers is offline?

This is where having some on-premises DNS servers comes in very handy. What we can essentially do here is send a sample of our users to our on-premises servers, get a baseline performance measurement, then break one of the servers and run the performance measurements again. We can also measure in multiple places: We have our measurements as reported by our clients (what the end user actually experienced), and we can look at data from within our network to see what actually happened. For network analysis, we turned to our trusted network analysis tool, ExtraHop. This would allow us to look at the data on the wire, and get measurements from a broken DNS server (something you can’t do easily with a pcap on that server, because, you know. It’s broken).

Here’s what healthy performance looked like on the wire (as measured by ExtraHop), with two DNS servers, both of them fully operational, over a 24-hour period (this chart is additive for the two series):

Blue and brown are the two different, healthy DNS servers. As you can see, there’s a very even 50:50 split in request volume. Because both of the servers are located in the same datacenter, Smoothed Round Trip Time had no effect, and we had a nice even distribution – as we would expect.

Now, what happens when we take one of those DNS servers offline, to simulate a provider outage?

In this case, the blue DNS server was offline, and the brown DNS server was healthy. What we see here is that the blue, broken, DNS server received the same number of requests as it did when the DNS server was healthy, but the brown, healthy, DNS server saw twice as many requests. This is because those users who were hitting the broken server eventually retried their requests to the healthy server and started to favor it. So what does this look like in terms of actual client performance?

I’m only going to share one chart with you this time, because they were all essentially the same:

What we see here is a substantial number of our visitors saw a performance decrease. For some it was minor, for others, quite major. This is because the 50% of visitors who hit the faulty server need to retry their request, and the amount of time it takes to retry that request seems to vary. You can see again a large increase in the long tail, which indicates that they are clients who took over 300 milliseconds to retry their request.

What does this tell us?

What this means is that in the event of a DNS provider going offline, we need to pull that DNS provider out of rotation to provide best performance, but until we do our users will still receive service. A non-trivial number of users will be seeing a large performance impact.

What is the best number of nameservers to be using?

Based on the previous performance testing, we can assume that the number of retries a client may have to make is N/2+1, where N is the number of nameservers listed. So if we list eight nameservers, with four from each provider, the client may potentially have to make 5 DNS requests before they finally get a successful message (the four failed requests, plus a final successful one). A statistician better than I would be able to tell you the exact probabilities of each scenario you would face, but the short answer here is:

Four.

We felt that based on our use case, and the performance penalty we were willing to take, we would be listing a total of four nameservers – two from each provider. This may not be the right decision for those who have a web presence orders of magnitudes larger than ours, but Facebook provide two nameservers on IPv4 and two on IPv6. Twitter provides eight, four from Dyn and four from Route 53. Google provides 4.

How are we going to keep our DNS providers in sync?

DNS has built in ways of keeping multiple servers in sync. You have domain transfers (IXFR, AXFR), which are usually triggered by a NOTIFY packet sent to all the servers listed as NS records in the zone. But these are not used in the wild very often, and have limited support from DNS providers. They also come with their own headaches, like maintaining an ACL IP Whitelist, of which there could be hundreds of potential servers (all the different points of presence from multiple providers), of which you do not control any. You also lose the ability to audit who changed which record, as they could be changed on any given server.

So we built a tool to keep our DNS in sync. We actually built this tool years ago, once our artisanally crafted zone files became too troublesome to edit by hand. The details of this tool are out of scope for this blog post though. If you want to learn about it, keep an eye out around March 2017 as we plan to open-source it. The tool lets us describe the DNS zone data in one place and push it to many different DNS providers.

So what did we learn?

The biggest takeaway from all of this, is that even if you have multiple DNS servers, DNS is still a single point of failure if they are all with the same provider and that provider goes offline. Until the Dyn attack this was pretty much “in theory” if you were using a large DNS provider, because until first the successful attack no large DNS provider had ever had an extended outage on all of its points of presence.

However, implementing multiple DNS providers is not entirely straightforward. There are performance considerations. You need to ensure that both of your zones are serving the same data. There can be such a thing as too many nameservers.

Lastly, we did all of this whilst following DNS best practices. We didn’t have to do any weird DNS trickery, or write our own DNS server to do non-standard things. When DNS was designed in 1987, I wonder if the authors knew the importance of what they were creating. I don’t know, but their design still stands strong and resilient today.

Attributions

• Thanks to Camelia Nicollet for her work in R to produce the graphs in this blog post

Let’s talk about encryption. Specifically, HTTPS encryption. If you’ve been following any of the U.S. election debates, encryption is a topic that the politicians want to talk about – but not in the way that most of us would like. And it’s not just exclusive to the U.S. – the U.K. is proposing banning encrypted services, Australia is similar. If you’re really into it, you can get information about most countries cryptography laws .

But one thing is very clear – if your traffic is not encrypted, it’s almost certainly being watched and monitored by someone in a government somewhere – this is the well publicised reason behind governments opposing widespread encryption. The NSA’s PRISM program is the most well known, which is also contributed to by the British and Australian intelligence agencies.

Which is why when the EFF announced their Let’s Encrypt project (in conjunction with Mozilla, Cisco, Akamai and others), we thought it sounded like a great idea.

The premise is simple:

• Provide free encryption certificates
• Make renewing certificates and installing them on your systems easy
• Keep the certificates secure by installing them properly and issuing them best practices
• Be transparent. Issued and revoked certificates are publically auditable
• Be open. Make a platform and a standard that anyone can use and build on.
• Benefit the internet through cooperation – don’t let one body control access to the service

Let’s Encrypt explain this elegantly themselves:

The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention.

The process goes a bit like this:

1. Get your web server up and running, as per normal, on HTTP.
2. Install the appropriate Let’s Encrypt tool for your platform. Currently there is ACME protocol support for:
   1. Apache (Let’s Encrypt)
   2. Nginx (Let’s Encrypt — experimental)
   3. HAProxy (janeczku)
   4. IIS (ACMESharp)
3. Run the tool. It will generate a Certificate Signing Request for your domain, submit it to Let’s Encrypt, and then give you options for validating the ownership of your domain. The easiest method of validating ownership is one that the tool can do automatically, which is creating a file with a pre-determined, random file name, that the Let’s Encrypt server can then validate
4. The tool then receives the valid certificate from the Let’s Encrypt Certificate Authority and installs it onto your systems, and configures your web server to use the certificate
5. You need to renew the certificate in fewer than 90 days – so you then need to set up a scheduled task (cron job for Linux, scheduled task for Windows) to execute the renewal command for your platform (see your tool’s documentation for this).

And that’s it. No copy/pasting your CSR into poorly built web interfaces, or waiting for the email to confirm the certificate to come through, or hand-building PEM files with certificate chains. No faxing documents to numbers in foreign countries. No panicking at the last minute because you forgot to renew your certificate. Free, unencumbered, automatically renewed, SSL certificates for life.

Who Let’s Encrypt is for

• People running their own web servers.
  • You could be small businesses running Windows SBS server
  • You could be a startup offering a Software as a Service platform
  • You could be a local hackerspace running a forum
  • You could be a highschool student with a website about making clocks
• People with a registered, publically accessible domain name
  • Let’s Encrypt requires some form of domain name validation, whether it be a file it can probe over HTTP to verify your ownership of the domain name, or creating a DNS record it can verify
  • Certificate Authorities no longer issue certificates for “made-up” internal domain names or reserved IP addresses

Who Let’s Encrypt is not for

• Anyone on shared web hosting
  • Let’s Encrypt requires the input of the server operator. If you are not running your own web server, then this isn’t for you.
• Anyone who wants to keep the existence of their certificates a secret
  • Every certificate issued by Let’s Encrypt is publically auditable, which means that if you don’t want anyone to know that you have a server on a given domain, then don’t use Let’s Encrypt
  • If you have sensitive server names (such as finance.corp.example.com), even though it’s firewalled, you might not want to use Let’s Encrypt
• Anyone who needs a wildcard certificate
  • Let’s Encrypt does not issue wildcard certificates. They don’t need to – they offer unlimited certificates, and you can even specify multiple Subject Alternative Names on your certificate signing request
  • However, you may still need a wildcard if:
    • You have a lot of domains and can’t use SNI (I’m looking at you, Android 2.x, of which there is still a non-trivial number of users)
    • You have systems that require a wildcard certificate (some unified communications systems do this)
• Anyone who needs a long-lived certificate
  • Let’s Encrypt certificates are only valid for 90 days, and must be renewed prior to then. If you need a long-lived certificate, then Let’s Encrypt is not for you
• Anyone who wants Extended Validation
  • Let’s Encrypt only validates that you have control over a given domain. It does not validate your identity or business or anything of that nature. As such you cannot get the green security bar that displays in the browser for places like banks or PayPal.
• Anyone who needs their certificate to be trusted by really old things
  • If you have devices from 1997 that only trust 1997’s list of CA’s, then you’re going to have a bad time
  • However, this is likely the least of your troubles
  • Let’s Encrypt is trusted by:
    • Android version 2.3.6 and above, released 2011-09-02
    • FireFox version 2.0 and above, released 2006-10-24
    • Internet Explorer on Windows Vista or above (For Windows XP, see this issue ), released 2007-01-30
    • Google Chrome on Windows Vista or above (For Windows XP, see this issue ), released 2008-08-02
    • Safari on OSX v4.0 or above (Mac OSX 10.4 or newer), released 2005-04-29
    • Safari on iOS v3.1 or above, released 2010-02-02

However, these are mostly edge cases, and if you’re reading this blog post, then you will know if they apply to you or not.

So let’s get out there and encrypt!

The elephant in the room

“But hang on!”, I hear the eagle-eyed reader say. “Stack Overflow is not using SSL/TLS!” you say. And you would be partly correct.

We do offer SSL on all our main sites. Go ahead, try it:

• https://stackoverflow.com/
• https://serverfault.com/
• https://meta.stackexchange.com/

However, we have some slightly more complicated issues at hand. For details about our issues, see the great blog post by Nick Craver. It’s from 2013 and we have fixed many of the issues that we were facing back then, but there is still some way to go.

However, all our signup and login pages however are delivered over HTTPS, and you can switch to HTTPS manually if you would prefer – for most sites.

Let’s get started

So how do you get started? If you have a debian-based Apache server, then grab the Let’s Encrypt tool and go!

If you’re on a different platform, then check the list of pre-build clients above, or take a look at a recent comparison of the most common *nix scripts.

---

 

Addendum: Michael Hampton pointed out to me that Fedora ships with the Let’s Encrypt package as a part of their distribution and is also in EPEL if you’re on RedHat, CentOS or another distribution that can make use of EPEL packages.

Craig Peterson joined us back in February. We neglected to announce him in a timely manner because we’re horrible people. That’s OK though, as he’s now one of us.

In contrast to the rest of the SRE team, Craig’s superpowers come in the form of development. He’s been focused on making our homegrown monitoring system, Bosun, better than all of the monitoring systems that have or will ever exist. As Bosun is open source, you can practically hover over Craig’s shoulder and watch his day to day work. In fact, I encourage you to. I’m sure Craig won’t mind in the least.

Craig hails from the state of Utah, making him our second Utahn (or Utahan) member of the SRE team.

---

Mark Henderson joins us from the nation of OZ. He has regaled us with tales of drop bears and his strong feelings about the suitability of Foster’s as a consumable liquid.

He has relocated nearly ten-thousand miles away to our New York office, and as such will finally get a slight taste of how December should feel. On the plus side, he will no longer have to contend with a smorgasbord of creatures which actively campaign to wipe out the human race.

Mark has also served as a Server Fault moderator since 2011, and has acquired quite a bit of rep in service of the community.

---

Please join me in welcoming Craig and Mark to the Site Reliability Engineering team!

A few weeks ago we upgraded a lot of the core infrastructure in our New York (okay, it’s really in New Jersey now – but don’t tell anyone) data center. We love being open with everything we do (including infrastructure), and really consider it one of the best job perks we have. So here’s how and why we upgrade a data center. First, take a moment to look at what Stack Overflow started as. It’s 5 years later and hardware has come a long way.

Why?

Up until 2 months ago, we hadn’t replaced any servers since upgrading from the original Stack Overflow web stack. There just hasn’t been a need since we first moved to the New York data center (Oct 23rd, 2010 – over 4 years ago). We’re always reorganizing, tuning, checking allocations, and generally optimizing code and infrastructure wherever we can. We mostly do this for page load performance; the lower CPU and memory usage on the web tier is usually a (welcomed) side-effect.

So what happened? We had a meetup. All of the Stack Exchange engineering staff got together at our Denver office in October last year and we made some decisions. One of those decisions was what to do about infrastructure hardware from a lifecycle and financial standpoint. We decided that from here on out: hardware is good for approximately 4 years. After that we will: retire it, replace it, or make an exception and extend the warranty on it. This lets us simplify a great many things from a management perspective, for example: we limit ourselves to 2 generations of servers at any given time and we aren’t in the warranty renewal business except for exceptions. We can order all hardware up front with the simple goal of 4 years of life and with a 4 year warranty.

Why 4 years? It seems pretty arbitrary. Spoiler alert: it is. We were running on 4 year old hardware at the time and it worked out pretty well so far. Seriously, that’s it: do what works for you. Most companies depreciate hardware across 3 years, making questions like “what do we do with the old servers?” much easier. For those unfamiliar, depreciated hardware effectively means “off the books.” We could re-purpose it outside production, donate it, let employees go nuts, etc. If you haven’t heard, we raised a little money recently. While the final amounts weren’t decided when we were at the company meetup in Denver, we did know that we wanted to make 2015 an investment year and beef up hardware for the next 4.

Over the next 2 months, we evaluated what was over 4 years old and what was getting close. It turns out almost all of our Dell 11th generation hardware (including the web tier) fits these criteria – so it made a lot of sense to replace the entire generation and eliminate a slew of management-specific issues with it. Managing just 12th and 13th generation hardware and software makes life a lot easier – and the 12th generation hardware will be mostly software upgradable to near equivalency to 13th gen around April 2015.

What Got Love

In those 2 months, we realized we were running on a lot of old servers (most of them from May 2010):

• Web Tier (11 servers)
• Redis Servers (2 servers)
• Second SQL Cluster (3 servers – 1 in Oregon)
• File Server
• Utility Server
• VM Servers (5 servers)
• Tag Engine Servers (2 servers)
• SQL Log Database

We also could use some more space, so let’s add on:

• An additional SAN
• An additional DAS for the backup server

That’s a lot of servers getting replaced. How many? This many:

The Upgrade

I know what you’re thinking: “Nick, how do you go about making such a fancy pile of servers?” I’m glad you asked. Here’s how a Stack Exchange infrastructure upgrade happens in the live data center. We chose not to failover for this upgrade; instead we used multiple points of redundancy in the live data center to upgrade it while all traffic was flowing from there.

Day -3 (Thursday, Jan 22nd): Our upgrade plan was finished (this took about 1.5 days total), including everything we could think of. We had limited time on-site, so to make the best of that we itemized and planned all the upgrades in advance (most of them successfully, read on). You can find a read the full upgrade plan here.

Day 0 (Sunday, Jan 25th): The on-site sysadmins for this upgrade were George Beech, Greg Bray, and Nick Craver (note: several remote sysadmins were heavily involved in this upgrade as well: Geoff Dalgas online from Corvallis, OR, Shane Madden, online from Denver, CO, and Tom Limoncelli who helped a ton with the planning online from New Jersey). Shortly before flying in we got some unsettling news about the weather. We packed our snow gear and headed to New York.

Day 1 (Monday, Jan 26th): While our office is in lower Manhattan, the data center is now located in Jersey City across the Hudson river: We knew there was a lot to get done in the time we had allotted in New York, weather or not. The thought was that if we skipped Monday we likely couldn’t get back to the data center Tuesday if the PATH (mass transit to New Jersey) shut down. This did end up happening. The team decision was: go time. We got overnight gear then headed to the data center. Here’s what was there waiting to be installed:

Yeah, we were pretty excited too. Before we got started with the server upgrade though, we first had to fix a critical issue with the redis servers supporting the launching-in-24-hours Targeted Job Ads. These machines were originally for Cassandra (we broke that data store), then Elasticsearch (broke that too), and eventually redis. Curious? Jason Punyon and Kevin Montrose have an excellent blog series on Providence, you can find Punyon’s post on what broke with each data store here.

The data drives we ordered for these then-redundant systems were the Samsung 840 Pro drives which turned out to have a critical firmware bug. This was causing our server-to-server copies across dual 10Gb network connections to top out around 12MB/s (ouch). Given the hundreds of gigs of memory in these redis instances, that doesn’t really work. So we needed to upgrade the firmware on these drives to restore performance. This needed to be online, letting the RAID 10 arrays rebuild as we went. Since you can’t really upgrade firmware over most USB interfaces, we tore apart this poor, poor little desktop to do our bidding:

Once that was kicked off, it ran in parallel with other work (since RAID 10s with data take tens of minutes to rebuild, even with SSDs). The end result was much improved 100-200MB/s file copies (we’ll see what new bottleneck we’re hitting soon – still lots of tuning to do). Now the fun begins. In Rack C (we have high respect for our racks, they get title casing), we wanted to move from the existing SFP+ 10Gb connectivity combined with 1Gb uplinks for everything else to a single dual 10Gb BASE-T (RJ45 connector) copper solution. This is for a few reasons: The SFP+ cabling we use is called twinaxial which is harder to work with in cable arms, has unpredictable girth when ordered, and can’t easily be gotten natively in the network daughter cards for these Dell servers. The SFP+ FEXes also don’t allow us to connect any 1Gb BASE-T items that we may have (though that doesn’t apply in this rack, it does when making it a standard across all racks like with our load balancers). So here’s what we started with in Rack C:

• 2 Nexus 2232PP 10Gb SFP+ FEXes at the top
• 2 Nexus 2248TP-E 1Gb BASE-T FEXes in the middle
• 1 2960 Management switch
• 2 Avocent KVM aggregator/uplinks
• 6U of Panduit cable management

What we want to end up with is:

• 2 Nexus 2232TM 10Gb BASE-T FEXes
• 1 2960 Management switch
• 1 Avocent KVM aggregator/uplink
• 5U of Panduit cable management

The plan was to simplify network config, cabling, overall variety, and save 4U in the process. Here’s what the top of the rack looked like when we started: …and the middle (cable management covers already off):

Let’s get started. First, we wanted the KVMs online while working so we, ummm, “temporarily relocated” them: Now that those are out of the way, it’s time to drop the existing SFP+ FEXes down as low as we could to install the new 10Gb BASE-T FEXes in their final home up top: The nature of how the Nexus Fabric Extenders work allows us to allocate between 1 and 8 uplinks to each FEX. This means we can unplug 4 ports from each FEX without any network interruption, take the 4 we find dead in the VPC (virtual port channel) out of the VPC and assign them to the new FEX. So we go from 8/0 to 4/4 to 0/8 overall as we move from old to new through the upgrade. Here’s the middle step of that process: With the new network in place, we can start replacing some servers. We yanked several old servers already, one we virtualized and 2 we didn’t need anymore. Combine this with evacuating our NY-VM01 & NY-VM02 hosts and we’ve made 5U of space through the rack. On top of NY-VM01&02 was 1 of the 1Gb FEXes and 1U of cable management. Luckily for us, everything is plugged into both FEXes and we could rip one out early. This means we could spin up the new VM infrastructure faster than we had planned. Yep, we’re already changing THE PLAN™. That’s how it goes. What are we replacing those aging VM servers with? I’m glad you asked. These bad boys:

There are 2 of these Dell PowerEdge FX2s Blade Chassis each with 2 FC630 blades. Each blade has dual Intel E5-2698v3 18-core processors and 768GB of RAM (and that’s only half capacity). Each chassis has 80Gbps of uplink capacity as well via the dual 4x 10Gb IOA modules. Here they are installed:

The split with 2 half-full chassis give us 2 things: capacity to expand by double, and avoiding any single points of failure with the VM hosts. That was easy, right? Well what we didn’t plan on was the network portion of the day, it turns out those IO Aggregators in the back are pretty much full switches with 4 external 10Gbps ports and 8 internal 10Gbps (2 per blade) ports each. Once we figured out what they could and couldn’t do, we got the bonding in place and the new hosts spun up.

It’s important to note here it wasn’t any of the guys in the data center spinning up this VM architecture after the network was live. We’re setup so that Shane Madden was able to do all this remotely. Once he had the new NY-VM01 & 02 online (now blades), we migrated all VMs over to those 2 hosts and were able to rip out the old NY-VM03-05 servers to make more room. As we ripped things out, Shane was able to spin up the last 2 blades and bring our new beasts fully online. The net result of this upgrade was substantially more CPU and memory (from 528GB to 3,072GB overall) as well as network connectivity. The old hosts each had 4x 1Gb (trunk) for most access and 2x 10Gb for iSCSI access to the SAN. The new blade hosts each have 20Gb of trunk access to all networks to split as they need.

But we’re not done yet. Here’s the new EqualLogic PS6210 SAN that went in below (that’s NY-LOGSQL01 further below going in as well):

Our old SAN was a PS6200 with 24x 900GB 10k drives and SFP+ only. This is a newer 10Gb BASE-T 24x 1.2TB 10k version with more speed, more space, and the ability to go active/active with the existing SAN. Along the the SAN we also installed this new NY-LOGSQL01 server (replacing an aging Dell R510 never designed to be a SQL server – it was purchased as a NAS):

The additional space freed by the other VM hosts let us install a new file and utility server:

Of note here: the NY-UTIL02 utility server has a lot of drive bays so we could install 8x Samsung 840 Pros in a RAID 0 in order to restore and test the SQL backups we make every night. It’s RAID 0 for space because all of the data is literally loaded from scratch nightly – there’s nothing to lose. An important lesson we learned last year was that the 840 Pros do not have capacitors in there and power loss will cause data loss if they’re active since they have a bit of DIMM for write cache on board. Given this info – we opted to stick some Intel S3700 800GB drives we had from the production SQL server upgrades into our NY-DEVSQL01 box and move the less resilient 840s to this restore server where it really doesn’t matter.

Okay, let’s snap back to blizzard reality. At this point mass transit had shut down and all hotels in (blizzard) walking distance were booked solid. Though we started checking accommodations as soon as we arrived on site, we had no luck finding any hotels. Though the blizzard did far less than predicted, it was still stout enough to shut everything down. So, we decided to go as late as we could and get ahead of schedule. To be clear: this was the decision of the guys on site, not management. At Stack Exchange employees are trusted to get things done, however they best perceive how to do that. It’s something we really love about this job.

If life hands you lemons, ignore those silly lemons and go install shiny new hardware instead.

This is where we have to give a shout out to our data center QTS. These guys had the office manager help us find any hotel we could, set out extra cots for us to crash on, and even ordered extra pizza and drinks so we didn’t go starving. This was all without asking – they are always fantastic and we’d recommend them to anyone looking for hosting in a heartbeat.

After getting all the VMs spun up, the SAN configured, and some additional wiring ripped out, we ended around 9:30am Tuesday morning when mass transit was spinning back up. To wrap up the long night, this was the near-heart attack we ended on, a machine locking up at: Turns out a power supply was just too awesome and needed replacing. The BIOS did successfully upgrade with the defective power supply removed and we got a replacement in before the week was done. Note: we ordered a new one rather than RMA the old one (which we did later). We keep a spare power supply for each wattage level in the data center, and try to use as few different levels as possible.

Day 2 (Tuesday, Jan 27th): We got some sleep, got some food, and arrived on site around 8pm. Starting the web tier (a rolling build out) was kicked off first:

While we rotated 3 servers at a time out for rebuilds on the new hardware, we also upgraded some existing R620 servers from 4x 1Gb network daughter cards to 2x 10Gb + 2x 1Gb NDCs. Here’s what that looks like for NY-SERVICE03:

The web tier rebuilding gave us a chance to clean up some cabling. Remember those 2 SFP+ FEXes? They’re almost empty: The last 2 items were the old SAN and that aging R510 NAS/SQL server. This is where the first major hiccup in our plan occurred. We planned to install a 3rd PCIe card in the backup server pictured here: We knew it was a Dell R620 10 bay chassis that has 3 half-height PCIe cards. We knew it had a SAS controller for the existing DAS and a PCIe card for the SFP+ 10Gb connections it has (it’s in the network rack with the cores in which all 96 ports are 10Gb SFP+). Oh hey look at that, it’s hooked to a tape drive which required another SAS controller we forgot about. Crap. Okay, these things happen. New plan.

We had extra 10Gb network daughter cards (NDCs) on hand, so we decided to upgrade the NDC in the backup server, remove the SFP+ PCIe card, and replace it with the new 12Gb SAS controller. We also forgot to bring the half-height mounting bracket for the new card and had to get creative with some metal snips (edit: turns out it never came with one – we feel slightly less dumb about this now). So how do we plug that new 10Gb BASE-T card into the network core? We can’t. At least not at 10Gb. Those 2 last SFP+ items in Rack C also need a home – so we decided to make a trade. The whole backup setup (including new MD1400 DAS) just love their new Rack C home:

Then we could finally remove those SFP+ FEXes, bring those KVMs back to sanity, and clean things up in Rack C:

See? There was a plan all along. The last item to go in Rack C for the day is NY-GIT02, our new Gitlab and TeamCity server:

Note: we used to run TeamCity on Windows on NY-WEB11. Geoff Dalgas threw out the idea during the upgrade of moving it to hardware: the NY-GIT02 box. Because they are such intertwined dependencies (for which both have an offsite backup), combining them actually made sense. It gave TeamCity more power, even faster disk access (it does a lot of XML file…stuff), and made the web tier more homogenous all at the same time. It also made the downtime of NY-WEB11 (which was imminent) have far less impact. This made lots of sense, so we changed THE PLAN™ and went with it. More specifically, Dalgas went with it and set it all up, remotely from Oregon. While this is happening, Greg was fighting with a DSC install hang regarding git on our web tier: Wow that’s a lot of red, I wonder who’s winning. And that’s Dalgas in a hangout on my laptop, hi Dalgas! Since the web tier builds were a relatively new process fighting us, we took the time to address some of the recent cabling changes. The KVMs were installed hastily not long before this because we knew a re-cable was coming. In Rack A for example we moved the top 10Gb FEX up a U to expand the cable management to 2U and added 1U of management space between the KVMs. Here’s that process:

Since we had to re-cable from the 1Gb middle FEXes in Rack A & B (all 4 being removed) to the 10Gb Top-of-Rack FEXes, we moved a few things around. The CloudFlare load balancers down below the web tier at the bottom moved up to spots freed by the recently virtualized DNS servers to join the other 2 public load balancers. The removal of the 1Gb FEXes as part of our all-10Gb overhaul meant that the middle of Racks A & B had much more space available, here’s the before and after:

After 2 batches of web servers, cable cleanup, and network gear removal, we called it quits around 8:30am to go grab some rest. Things were moving well and we only had half the web tier, cabling, and a few other servers left to replace.

Day 3 (Wednesday, Jan 28th): We were back in the data center just before 5pm, set up and ready to go. The last non-web servers to be replaced were the redis and “service” (tag engine, elasticsearch indexing, etc.) boxes:

We have 3 tag engine boxes (purely for reload stalls and optimal concurrency, not load) and 2 redis servers in the New York data center. One of the tag engine boxes was a more-recent R620, (this one got the 10Gb upgrade earlier) and wasn’t replaced. That left NY-SERVICE04, NY-SERVICE05, NY-REDIS01 and NY-REDIS02. On the service boxes the process was pretty easy, though we did learn something interesting: if you put both of the drives from the RAID 10 OS array in an R610 into the new R630…it boots all the way into Windows 2012 without any issues. This threw us for a moment because we didn’t remember building it in the last 3 minutes. Rebuild is simple: lay down Windows 2012 R2 via our image + updates + DSC, then install the jobs they do. StackServer (from a sysadmin standpoint) is simply a windows service – and our TeamCity build handles the install and such, it’s literally just a parameter flag. These boxes also run a small IIS instance for internal services but that’s also a simple build out. The last task they do is host a DFS share, which we wanted to trim down and simplify the topology of, so we left them disabled as DFS targets and tackled that the following week – we had NY-SERVICE03 in rotation for the shares and could do such work entirely remotely. For redis we always have a slave chain happening, it looks like this: This means we can do an upgrade/failover/upgrade without interrupting service at all. After all those buildouts, here’s the super fancy new web tier installed:

To get an idea of the scale of hardware difference, the old web tier was Dell R610s with dual Intel E5640 processors and 48GB of RAM (upgraded over the years). The new web tier has dual Intel 2687W v3 processors and 64GB of DDR4 memory. We re-used the same dual Intel 320 300GB SSDs for the OS RAID 1. If you’re curious about specs on all this hardware – the next post we’ll do is a detailed writeup of our current infrastructure including exact specs.

Day 4 (Thursday, Jan 29th): I picked a fight with the cluster rack, D. Much of the day was spent giving the cluster rack a makeover now that we had most of the cables we needed in. When it was first racked, the pieces we needed hadn’t arrived by go time. It turns out we were still short a few cat and power cables as you’ll see in the photos, but we were able to get 98% of the way there.

It took a while to whip this rack into shape because we added cable arms where they were missing, replaced most of the cabling, and are fairly particular about the way we do things. For instance: how do you know things are plugged into the right port and where the other end of the cable goes? Labels. Lots and lots of labels. We label both ends of every cable and every server on both sides. It adds a bit of time now, but it saves both time and mistakes later.

Here’s what the racks ended up looking like when we ran out of time this trip:

It’s not perfect since we ran out of several cables of the proper color and length. We have ordered those and George will be tidying the last few bits up.

I know what you’re thinking. We don’t think that’s enough server eye-candy either.

Here’s the full album of our move.

And here’s the #SnowOps twitter stream which has a bit more.

What Went Wrong

• We’d be downright lying to say everything went smoothly. Hardware upgrades of this magnitude never do. Expect it. Plan for it. Allow time for it.
• Remember when we upgraded to those new database servers in 2010 and the performance wasn’t what we expected? Yeah, that. There is a bug we’re currently helping Dell track down in their 1.0.4/1.1.4 BIOS for these systems that seems to not respect whatever performance setting you have. With Windows, a custom performance profile disabling C-States to stay at max performance works. In CentOS 7, it does not – but disabling the Intel PState driver does. We have even ordered and just racked a minimal R630 to test and debug issues like this as well as test our deployment from bare metal to constantly improve our build automation. Whatever is at fault with these settings not being respected, our goal is to get that vendor to release an update addressing the issue so that others don’t get the same nasty surprise.
• We ran into an issue deploying our web tier with DSC getting locked up on a certain reboot thinking it needed a reboot to finish but coming up in the same state after a reboot in an endless cycle. We also hit issues with our deployment of the git client on those machines.
• We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.
• We learned that if you move the drives from a RAID array from an R610 to an R630 and don’t catch the PXE boot prompt, the server will happily boot all the way into the OS.
• We learned the good and the bad of the Dell FX2 IOA architecture and how they are self-contained switches.
• We learned the CMC (management) ports on the FX2 chassis are effectively a switch. We knew they were suitable for daisy chaining purposes. However, we promptly forgot this, plugged them both in for redundancy and created a switching loop that reset Spanning Tree on our management network. Oops.
• We learned the one guy on twitter who was OCD about the one upside down box was right. It was a pain to flip that web server over after opening it upside down and removing some critical box supports.
• We didn’t mention this was a charge-only cable. Wow, that one riled twitter up. We appreciate the #infosec concern though!
• We drastically underestimated how much twitter loves naked servers. It’s okay, we do too.
• We learned that Dell MD1400 (13g and 12Gb/s) DAS (direct attached storage) arrays do not support hooking into their 12g servers like our R620 backup server. We’re working with them on resolving this issue.
• We learned Dell hardware diagnostics don’t even check the power supply, even when the server has an orange light on the front complaining about it.
• We learned that Blizzards are cold, the wind is colder, and sleep is optional.

The Payoff

Here’s what the average render time for question pages looks like, if you look really closely you can guess when the upgrade happened: The decrease on question render times (from approx 30-35ms to 10-15ms) is only part of the fun. The next post in this series will detail many of the other drastic performance increases we’ve seen as the result of our upgrades. Stay tuned for a lot of real world payoffs we’ll share in the coming weeks.

Does all this sound like fun?

To us, it is fun. If you feel the same way, come do it with us. We are specifically looking for sysadmins preferably with data center experience to come help out in New York. We are currently hiring 2 positions:

• Site Reliability Engineer: Generalist
• Site Reliability Engineer: Networking

If you’re curious at all, please ask us questions here, Twitter, or wherever you’re most comfortable. Really. We love Q&A.

The Site Reliability Engineering team at Stack Exchange is excited to announce our latest addition – Jason Harvey!

Jason hails from Alaska and although he cannot see Russia from his house, he is officially our most Northwestern US Employee beating Geoff Dalgas by over 1500 miles. He will be working remotely except when we decide to fly him to the mainland to help move some servers or sample the excellent work of the NYC Stack Exchange Chefs.

Jason brings a wealth of knowledge to our team from his previous roles working at reddit and Rackspace, and he has already been extremely helpful in our ongoing implementation of CloudFlare. He also has a deep understanding of the complex issues that surround our industry and shares our belief of being open and transparent whenever possible.

We believe Jason is an excellent addition to our SRE team so I hope you will join me in welcoming him!

Imagine if alerting was what you wanted it to be:

• Every alert you received was actionable, and there were few false alerts
• Notifications were actually informative
• You received alerts in time to fix problems before they impacted your users

This isn’t the world we live in…

• We accept lots of notifications from our alerting system that are not actionable
• The notifications don’t tell us about the problem
• We get paged when stuff is dead and not when it is sick

In order to resolve the dissonance between reality and what alerting should be we need:

• An expressive way to evaluate alert conditions that isn’t a 1:1 mapping to the metrics
• Alerts backed by time-series and not just recent values
• A way to to make rich notifications that include useful information
• A way to iterate fast with alert design so that our alerts are continuously improved

A little less than a year ago, Matt Jibson and Kyle Brandt set out to create a system to solve this and other problems in monitoring; we call it Bosun. Our belief is that achieving excellence in alerting is a complex problem and requires a powerful and flexible platform to design alerts. Therefore, Bosun’s strategy is to provide a framework that enables the operator to create intelligent and informative alerts. We believe that you are smarter and more creative than any monitoring system can be when it comes to your environment.

In order to achieve that, at the highest level Bosun provides:

• An expression language (a small domain-specific language) designed to allow for the creation of highly flexible and specific alerts
• Notification templates that allow you to include whatever information you think is relevant
• A web interface the provides a workflow for more rapid iteration with improving and creating alerts: Graph -> Expression -> Rule + Template -> Test the rule over history

The Expression Language

We believe that every alert requires action. An alert asks for your attention, and human attention and time is a valuable asset. So alerting is about owning the operators attention. Taking action with alerts practically means one of two things. If the alert was accurate, then you fix the issue that triggered the alert. If the alert was a false positive, then the alert should be tuned in a way that the false positive won’t trigger the alert. This is where things tend to fall down because alert evaluations are not powerful enough to be tuned. With Bosun’s expression language, you can tune alerts in the following ways:

• Alert thresholds based on history vs static thresholds (or both combined)
• Statistics functions: Min, Percentile, Median, Deviations, Forecasting. You can change the duration that these evaluate over (i.e. 5 minutes, 1 hour, 1 week?)
• Scope-aware: How should components in your environment be grouped? By Host, subsystem, cluster, a combination of those things
• Boolean conditions: The interaction of multiple components

These possibilities, when applied selectively by a skilled operator, provide ample ways to reduce alerting noise.

Notification Templates

Once you have someone’s attention with a valid alert, you need to direct them to the problem as accurately as possible. Our notification templates use the Go template language, which means they can be quite flexible. Notifications in Bosun allow you to:

• Include breakdowns of information related to your alert as embedded graphs, html tables, or whatever else you think makes sense
• Include information that wasn’t directly related to the alert: i.e. CPU of a host even though it was a memory alert
• Generate links to your dashboards or other sources of information
• Includes notes about why you created that alert, caveats, and other information the person being notified should be aware of

The Workflow

One of the main issues with alerting is that there is so much friction to tuning alerts that it doesn’t get done. One of Bosun’s goals was to provide a faster iteration cycle for creating and tuning alerts by making the web interface an alerting IDE: Graphs in Bosun’s interface link to expressions, which then link to alert rules and templates. You can then test alerts before implementing; the results of a rule and template can be tested in the interface. You can test how they will behave currently, how they might have behaved at a past time, or generate a timeline of how they might have behaved over a range of time.

This means that your alert tuning doesn’t need to be totally reactionary. You can test alert changes and see how and when they would have triggered over the past weeks (or longer, if you are patient). This results in less alert noise being sent to operators.

But wait! There’s more!

Bosun has also attempted to make some problems in monitoring easier:

• Getting data into the system: our agent (called “scollector”) runs on Windows and Linux and starts sending data to Bosun
• Applications can push metrics to the system via JSON API calls
• Human maintenance: Properly designed alerts will apply to new systems, and services are auto-discovered by scollector. This means you don’t have to remember to update your monitoring most of the time when a new services and hosts are deployed (as long as scollector is pushed out via your build or configuration management process)

We hope you go try this out. We have a docker image that has everything you need—just follow the getting started guide. We hope Bosun is useful to the community. We need your creativity and ideas to continue to grow it (and some contributors would be nice too!). We owe a special thanks to everyone else at Stack Exchange for:

• Contributing to scollector – Greg Bray has been working hard to fill out our Windows metrics, and Sam Torno did the same for Linux
• Getting a docker build – Peter Grace (who also did a lot of the dogfooding)
• Manning the front lines to keep the site up while we built this – the rest of the SRE Team
• Feature ideas and monitoring concepts – Tom Limoncelli and his monitoring chapters in The Practice of Cloud System Administration
• Letting Matt and I go tilting at windmills – Stack Exchange, Inc.

The Site Reliability Engineering team at Stack Exchange has a new addition – Greg Bray!

Greg joins us as our new Windows-focused generalist (in case you missed it, our friend Steven Murawski moved on recently), though he’s happy to work on whatever technology we throw at him. He’s a software developer turned sysadmin, and he’ll be assisting us in our quest to automate our infrastructure until it achieves sentience.

Greg is a participant on a number of the sites in the Stack Exchange network (since the early days in 2008!), a University of Utah Computer Engineering graduate (where he worked on a FPGA-based tester for NAND flash storage [pdf link]), and an occasional blogger. He has a knack for finding obscure bugs.

Greg lives in Salt Lake City with his wife and 4 year old cat named Kitty, and he’ll be working remotely from there. When he’s not working with technology, he enjoys biking, camping, and golfing.

Join me in welcoming Greg to our team!

The ServerFault Systems Administration team continues its growth with the addition of sysadmin Shane Madden. Shane lives in Denver and will work out of the Denver office on days he wants lunch.

You may know Shane already from his contributions in the open source world. He’s an avid Puppet programmer and Python coder. Shane’s very active on Server Fault… he has over 69k reputation points which makes him the 7th highest rep user there!

Shane’s hobbies include skiing, board games, video games and hockey. He has many pets including a mantis shrimp.

Please join me in welcoming Shane to the team!

Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.

How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.

Observability is the Foundation

Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:

• Questions operators have about their systems can be quantifiably answered with minimal effort
• Operators have rich mental models of how their systems function

When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.

Decision Making and Incident Preparedness

Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.

OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.

Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.

In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.

You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.

Other Benefits

Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.

Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.

Convinced? 5 Steps to Achieving Good Observability:

In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.

1. Instrument your systems by publishing metrics and events
2. Gather those metrics and events in a queryable data store(s)
3. Make that data readily accessible
4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
5. Establish the resources to drill down into abnormal or out of bounds behavior

Each of these steps largely depends on the previous step to be successful.

1. Instrument your Systems

Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:

1. Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
2. Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.

Good Metrics also tend to have these properties:

• High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
• Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
• Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.

It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.

2. Gather those metrics in a queryable data store(s)

This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:

• Building dashboards
• Enabling capacity planning
• Allowing operators to explore the data and learn
• Allowing people to invent cool stuff you didn’t anticipate

As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:

• Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
• Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views

3. Make that data Readily Accessible

If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:

• A fast responsive UI to allow for operators to drill down and explore easily
• Enables operators to create their own dashboards and graphs
• Highlight problems

4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior

Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.

Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.

Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.

5. Establish the resources to drill down into abnormal or out of bounds behavior

The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:

1. Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
2. Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
3. Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)

Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.

Use the science, Luke

If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.