A few weeks ago we upgraded a lot of the core infrastructure in our New York (okay, it’s really in New Jersey now – but don’t tell anyone) data center. We love being open with everything we do (including infrastructure), and really consider it one of the best job perks we have. So here’s how and why we upgrade a data center. First, take a moment to look at what Stack Overflow started as. It’s 5 years later and hardware has come a long way.


Up until 2 months ago, we hadn’t replaced any servers since upgrading from the original Stack Overflow web stack. There just hasn’t been a need since we first moved to the New York data center (Oct 23rd, 2010 – over 4 years ago).  We’re always reorganizing, tuning, checking allocations, and generally optimizing code and infrastructure wherever we can. We mostly do this for page load performance; the lower CPU and memory usage on the web tier is usually a (welcomed) side-effect.

So what happened? We had a meetup. All of the Stack Exchange engineering staff got together at our Denver office in October last year and we made some decisions. One of those decisions was what to do about infrastructure hardware from a lifecycle and financial standpoint. We decided that from here on out: hardware is good for approximately 4 years. After that we will: retire it, replace it, or make an exception and extend the warranty on it. This lets us simplify a great many things from a management perspective, for example: we limit ourselves to 2 generations of servers at any given time and we aren’t in the warranty renewal business except for exceptions. We can order all hardware up front with the simple goal of 4 years of life and with a 4 year warranty.

Why 4 years? It seems pretty arbitrary. Spoiler alert: it is. We were running on 4 year old hardware at the time and it worked out pretty well so far. Seriously, that’s it: do what works for you. Most companies depreciate hardware across 3 years, making questions like “what do we do with the old servers?” much easier. For those unfamiliar, depreciated hardware effectively means “off the books.” We could re-purpose it outside production, donate it, let employees go nuts, etc. If you haven’t heard, we raised a little money recently. While the final amounts weren’t decided when we were at the company meetup in Denver, we did know that we wanted to make 2015 an investment year and beef up hardware for the next 4.

Over the next 2 months, we evaluated what was over 4 years old and what was getting close. It turns out almost all of our Dell 11th generation hardware (including the web tier) fits these criteria – so it made a lot of sense to replace the entire generation and eliminate a slew of management-specific issues with it. Managing just 12th and 13th generation hardware and software makes life a lot easier – and the 12th generation hardware will be mostly software upgradable to near equivalency to 13th gen around April 2015.

What Got Love

In those 2 months, we realized we were running on a lot of old servers (most of them from May 2010):

  • Web Tier (11 servers)
  • Redis Servers (2 servers)
  • Second SQL Cluster (3 servers – 1 in Oregon)
  • File Server
  • Utility Server
  • VM Servers (5 servers)
  • Tag Engine Servers (2 servers)
  • SQL Log Database

We also could use some more space, so let’s add on:

  • An additional SAN
  • An additional DAS for the backup server

That’s a lot of servers getting replaced. How many? This many: Greg Bray, and a lot of old servers.

The Upgrade

I know what you’re thinking: “Nick, how do you go about making such a fancy pile of servers?” I’m glad you asked. Here’s how a Stack Exchange infrastructure upgrade happens in the live data center. We chose not to failover for this upgrade; instead we used multiple points of redundancy in the live data center to upgrade it while all traffic was flowing from there.

Day -3 (Thursday, Jan 22nd): Our upgrade plan was finished (this took about 1.5 days total), including everything we could think of. We had limited time on-site, so to make the best of that we itemized and planned all the upgrades in advance (most of them successfully, read on). You can find a read the full upgrade plan here.

Day 0 (Sunday, Jan 25th): The on-site sysadmins for this upgrade were George Beech, Greg Bray, and Nick Craver (note: several remote sysadmins were heavily involved in this upgrade as well: Geoff Dalgas online from Corvallis, OR, Shane Madden, online from Denver, CO, and Tom Limoncelli who helped a ton with the planning online from New Jersey). Shortly before flying in we got some unsettling news about the weather. We packed our snow gear and headed to New York.

Day 1 (Monday, Jan 26th): While our office is in lower Manhattan, the data center is now located in Jersey City across the Hudson river: Office to the Data Center We knew there was a lot to get done in the time we had allotted in New York, weather or not. The thought was that if we skipped Monday we likely couldn’t get back to the data center Tuesday if the PATH (mass transit to New Jersey) shut down. This did end up happening. The team decision was: go time. We got overnight gear then headed to the data center. Here’s what was there waiting to be installed:

Web, Redis, and Service serversNew 10Gb network gearFX2s Blade Chassis for VMs

Yeah, we were pretty excited too. Before we got started with the server upgrade though, we first had to fix a critical issue with the redis servers supporting the launching-in-24-hours Targeted Job Ads. These machines were originally for Cassandra (we broke that data store), then Elasticsearch (broke that too), and eventually redis. Curious? Jason Punyon and Kevin Montrose have an excellent blog series on Providence, you can find Punyon’s post on what broke with each data store here.

The data drives we ordered for these then-redundant systems were the Samsung 840 Pro drives which turned out to have a critical firmware bug. This was causing our server-to-server copies across dual 10Gb network connections to top out around 12MB/s (ouch). Given the hundreds of gigs of memory in these redis instances, that doesn’t really work. So we needed to upgrade the firmware on these drives to restore performance. This needed to be online, letting the RAID 10 arrays rebuild as we went. Since you can’t really upgrade firmware over most USB interfaces, we tore apart this poor, poor little desktop to do our bidding:

Once that was kicked off, it ran in parallel with other work (since RAID 10s with data take tens of minutes to rebuild, even with SSDs). The end result was much improved 100-200MB/s file copies (we’ll see what new bottleneck we’re hitting soon – still lots of tuning to do). Now the fun begins. In Rack C (we have high respect for our racks, they get title casing), we wanted to move from the existing SFP+ 10Gb connectivity combined with 1Gb uplinks for everything else to a single dual 10Gb BASE-T (RJ45 connector) copper solution. This is for a few reasons: The SFP+ cabling we use is called twinaxial which is harder to work with in cable arms, has unpredictable girth when ordered, and can’t easily be gotten natively in the network daughter cards for these Dell servers. The SFP+ FEXes also don’t allow us to connect any 1Gb BASE-T items that we may have (though that doesn’t apply in this rack, it does when making it a standard across all racks like with our load balancers). So here’s what we started with in Rack C:

What we want to end up with is:

The plan was to simplify network config, cabling, overall variety, and save 4U in the process. Here’s what the top of the rack looked like when we started: …and the middle (cable management covers already off):

Let’s get started. First, we wanted the KVMs online while working so we, ummm, “temporarily relocated” them: Now that those are out of the way, it’s time to drop the existing SFP+ FEXes down as low as we could to install the new 10Gb BASE-T FEXes in their final home up top: The nature of how the Nexus Fabric Extenders work allows us to allocate between 1 and 8 uplinks to each FEX. This means we can unplug 4 ports from each FEX without any network interruption, take the 4 we find dead in the VPC (virtual port channel) out of the VPC and assign them to the new FEX. So we go from 8/0 to 4/4 to 0/8 overall as we move from old to new through the upgrade. Here’s the middle step of that process: With the new network in place, we can start replacing some servers. We yanked several old servers already, one we virtualized and 2 we didn’t need anymore. Combine this with evacuating our NY-VM01 & NY-VM02 hosts and we’ve made 5U of space through the rack. On top of NY-VM01&02 was 1 of the 1Gb FEXes and 1U of cable management. Luckily for us, everything is plugged into both FEXes and we could rip one out early. This means we could spin up the new VM infrastructure faster than we had planned. Yep, we’re already changing THE PLAN™. That’s how it goes. What are we replacing those aging VM servers with? I’m glad you asked. These bad boys:

There are 2 of these Dell PowerEdge FX2s Blade Chassis each with 2 FC630 blades. Each blade has dual Intel E5-2698v3 18-core processors and 768GB of RAM (and that’s only half capacity). Each chassis has 80Gbps of uplink capacity as well via the dual 4x 10Gb IOA modules. Here they are installed:

The split with 2 half-full chassis give us 2 things: capacity to expand by double, and avoiding any single points of failure with the VM hosts. That was easy, right? Well what we didn’t plan on was the network portion of the day, it turns out those IO Aggregators in the back are pretty much full switches with 4 external 10Gbps ports and 8 internal 10Gbps (2 per blade) ports each. Once we figured out what they could and couldn’t do, we got the bonding in place and the new hosts spun up.

It’s important to note here it wasn’t any of the guys in the data center spinning up this VM architecture after the network was live. We’re setup so that Shane Madden was able to do all this remotely. Once he had the new NY-VM01 & 02 online (now blades), we migrated all VMs over to those 2 hosts and were able to rip out the old NY-VM03-05 servers to make more room. As we ripped things out, Shane was able to spin up the last 2 blades and bring our new beasts fully online. The net result of this upgrade was substantially more CPU and memory (from 528GB to 3,072GB overall) as well as network connectivity. The old hosts each had 4x 1Gb (trunk) for most access and 2x 10Gb for iSCSI access to the SAN. The new blade hosts each have 20Gb of trunk access to all networks to split as they need.

But we’re not done yet. Here’s the new EqualLogic PS6210 SAN that went in below (that’s NY-LOGSQL01 further below going in as well):

VM Servers, SAN, and NY-LOGSQL01 Our old SAN was a PS6200 with 24x 900GB 10k drives and SFP+ only. This is a newer 10Gb BASE-T 24x 1.2TB 10k version with more speed, more space, and the ability to go active/active with the existing SAN. Along the the SAN we also installed this new NY-LOGSQL01 server (replacing an aging Dell R510 never designed to be a SQL server – it was purchased as a NAS):

The additional space freed by the other VM hosts let us install a new file and utility server:

Of note here: the NY-UTIL02 utility server has a lot of drive bays so we could install 8x Samsung 840 Pros in a RAID 0 in order to restore and test the SQL backups we make every night. It’s RAID 0 for space because all of the data is literally loaded from scratch nightly – there’s nothing to lose. An important lesson we learned last year was that the 840 Pros do not have capacitors in there and power loss will cause data loss if they’re active since they have a bit of DIMM for write cache on board. Given this info – we opted to stick some Intel S3700 800GB drives we had from the production SQL server upgrades into our NY-DEVSQL01 box and move the less resilient 840s to this restore server where it really doesn’t matter.

Okay, let’s snap back to blizzard reality. At this point mass transit had shut down and all hotels in (blizzard) walking distance were booked solid. Though we started checking accommodations as soon as we arrived on site, we had no luck finding any hotels. Though the blizzard did far less than predicted, it was still stout enough to shut everything down. So, we decided to go as late as we could and get ahead of schedule. To be clear: this was the decision of the guys on site, not management. At Stack Exchange employees are trusted to get things done, however they best perceive how to do that. It’s something we really love about this job.

If life hands you lemons, ignore those silly lemons and go install shiny new hardware instead.

This is where we have to give a shout out to our data center QTS. These guys had the office manager help us find any hotel we could, set out extra cots for us to crash on, and even ordered extra pizza and drinks so we didn’t go starving. This was all without asking – they are always fantastic and we’d recommend them to anyone looking for hosting in a heartbeat.

After getting all the VMs spun up, the SAN configured, and some additional wiring ripped out, we ended around 9:30am Tuesday morning when mass transit was spinning back up. To wrap up the long night, this was the near-heart attack we ended on, a machine locking up at: BIOS Lockup Turns out a power supply was just too awesome and needed replacing. The BIOS did successfully upgrade with the defective power supply removed and we got a replacement in before the week was done. Note: we ordered a new one rather than RMA the old one (which we did later). We keep a spare power supply for each wattage level in the data center, and try to use as few different levels as possible.

Day 2 (Tuesday, Jan 27th): We got some sleep, got some food, and arrived on site around 8pm. Starting the web tier (a rolling build out) was kicked off first:

A stack of web serversA line of web serversSame line!Inside a web server

While we rotated 3 servers at a time out for rebuilds on the new hardware, we also upgraded some existing R620 servers from 4x 1Gb network daughter cards to 2x 10Gb + 2x 1Gb NDCs. Here’s what that looks like for NY-SERVICE03:

A line of web serversSame line!Inside a web server

The web tier rebuilding gave us a chance to clean up some cabling. Remember those 2 SFP+ FEXes? They’re almost empty: The last 2 items were the old SAN and that aging R510 NAS/SQL server. This is where the first major hiccup in our plan occurred. We planned to install a 3rd PCIe card in the backup server pictured here: We knew it was a Dell R620 10 bay chassis that has 3 half-height PCIe cards. We knew it had a SAS controller for the existing DAS and a PCIe card for the SFP+ 10Gb connections it has (it’s in the network rack with the cores in which all 96 ports are 10Gb SFP+). Oh hey look at that, it’s hooked to a tape drive which required another SAS controller we forgot aboutCrap. Okay, these things happen. New plan.

We had extra 10Gb network daughter cards (NDCs) on hand, so we decided to upgrade the NDC in the backup server, remove the SFP+ PCIe card, and replace it with the new 12Gb SAS controller. We also forgot to bring the half-height mounting bracket for the new card and had to get creative with some metal snips (edit: turns out it never came with one – we feel slightly less dumb about this now). So how do we plug that new 10Gb BASE-T card into the network core? We can’t. At least not at 10Gb. Those 2 last SFP+ items in Rack C also need a home – so we decided to make a trade. The whole backup setup (including  new MD1400 DAS) just love their new Rack C home:

Then we could finally remove those SFP+ FEXes, bring those KVMs back to sanity, and clean things up in Rack C:

Those pesky hanging KVMsTop of Rack CMiddle of Rack C

See? There was a plan all along. The last item to go in Rack C for the day is NY-GIT02, our new Gitlab and TeamCity server:

Signatures from the New York devsRacked and ready to go

Note: we used to run TeamCity on Windows on NY-WEB11. Geoff Dalgas threw out the idea during the upgrade of moving it to hardware: the NY-GIT02 box. Because they are such intertwined dependencies (for which both have an offsite backup), combining them actually made sense. It gave TeamCity more power, even faster disk access (it does a lot of XML file…stuff), and made the web tier more homogenous all at the same time. It also made the downtime of NY-WEB11 (which was imminent) have far less impact. This made lots of sense, so we changed THE PLAN™ and went with it. More specifically, Dalgas went with it and set it all up, remotely from Oregon. While this is happening, Greg was fighting with a DSC install hang regarding git on our web tier: Greg losing to DSC Wow that’s a lot of red, I wonder who’s winning. And that’s Dalgas in a hangout on my laptop, hi Dalgas! Since the web tier builds were a relatively new process fighting us, we took the time to address some of the recent cabling changes. The KVMs were installed hastily not long before this because we knew a re-cable was coming. In Rack A for example we moved the top 10Gb FEX up a U to expand the cable management to 2U and added 1U of management space between the KVMs. Here’s that process:

A messy starting KVMRemoving the cable management to make roomAhhhh room!That's better, all done.

Since we had to re-cable from the 1Gb middle FEXes in Rack A & B (all 4 being removed) to the 10Gb Top-of-Rack FEXes, we moved a few things around. The CloudFlare load balancers down below the web tier at the bottom moved up to spots freed by the recently virtualized DNS servers to join the other 2 public load balancers. The removal of the 1Gb FEXes as part of our all-10Gb overhaul meant that the middle of Racks A & B had much more space available, here’s the before and after:

Web tier below a 1Gb FEXLook at all that space!

After 2 batches of web servers, cable cleanup, and network gear removal, we called it quits around 8:30am to go grab some rest. Things were moving well and we only had half the web tier, cabling, and a few other servers left to replace.

Day 3 (Wednesday, Jan 28th): We were back in the data center just before 5pm, set up and ready to go. The last non-web servers to be replaced were the redis and “service” (tag engine, elasticsearch indexing, etc.) boxes:

A look inside redisNY-REDIS01 and NY-SERVICE05 racked and ready for an OS

We have 3 tag engine boxes (purely for reload stalls and optimal concurrency, not load) and 2 redis servers in the New York data center. One of the tag engine boxes was a more-recent R620, (this one got the 10Gb upgrade earlier) and wasn’t replaced. That left NY-SERVICE04, NY-SERVICE05, NY-REDIS01 and NY-REDIS02. On the service boxes the process was pretty easy, though we did learn something interesting: if you put both of the drives from the RAID 10 OS array in an R610 into the new R630…it boots all the way into Windows 2012 without any issues. This threw us for a moment because we didn’t remember building it in the last 3 minutes. Rebuild is simple: lay down Windows 2012 R2 via our image + updates + DSC, then install the jobs they do. StackServer (from a sysadmin standpoint) is simply a windows service – and our TeamCity build handles the install and such, it’s literally just a parameter flag. These boxes also run a small IIS instance for internal services but that’s also a simple build out. The last task they do is host a DFS share, which we wanted to trim down and simplify the topology of, so we left them disabled as DFS targets and tackled that the following week – we had NY-SERVICE03 in rotation for the shares and could do such work entirely remotely. For redis we always have a slave chain happening, it looks like this: This means we can do an upgrade/failover/upgrade without interrupting service at all. After all those buildouts, here’s the super fancy new web tier installed:

To get an idea of the scale of hardware difference, the old web tier was Dell R610s with dual Intel E5640 processors and 48GB of RAM (upgraded over the years). The new web tier has dual Intel 2687W v3 processors and 64GB of DDR4 memory. We re-used the same dual Intel 320 300GB SSDs for the OS RAID 1. If you’re curious about specs on all this hardware – the next post we’ll do is a detailed writeup of our current infrastructure including exact specs.

Day 4 (Thursday, Jan 29th): I picked a fight with the cluster rack, D. Much of the day was spent giving the cluster rack a makeover now that we had most of the cables we needed in. When it was first racked, the pieces we needed hadn’t arrived by go time. It turns out we were still short a few cat and power cables as you’ll see in the photos, but we were able to get 98% of the way there.

It took a while to whip this rack into shape because we added cable arms where they were missing, replaced most of the cabling, and are fairly particular about the way we do things. For instance: how do you know things are plugged into the right port and where the other end of the cable goes? Labels. Lots and lots of labels. We label both ends of every cable and every server on both sides. It adds a bit of time now, but it saves both time and mistakes later.

Cable labels!Web servers without labelsWeb servers with labels!Web server rear labels

Here’s what the racks ended up looking like when we ran out of time this trip:

It’s not perfect since we ran out of several cables of the proper color and length. We have ordered those and George will be tidying the last few bits up.

I know what you’re thinking. We don’t think that’s enough server eye-candy either.

Here’s the full album of our move.

And here’s the #SnowOps twitter stream which has a bit more.

What Went Wrong

  • We’d be downright lying to say everything went smoothly. Hardware upgrades of this magnitude never do. Expect it. Plan for it. Allow time for it.
  • Remember when we upgraded to those new database servers in 2010 and the performance wasn’t what we expected? Yeah, that. There is a bug we’re currently helping Dell track down in their 1.0.4/1.1.4 BIOS for these systems that seems to not respect whatever performance setting you have. With Windows, a custom performance profile disabling C-States to stay at max performance works. In CentOS 7, it does not – but disabling the Intel PState driver does. We have even ordered and just racked a minimal R630 to test and debug issues like this as well as test our deployment from bare metal to constantly improve our build automation. Whatever is at fault with these settings not being respected, our goal is to get that vendor to release an update addressing the issue so that others don’t get the same nasty surprise.
  • We ran into an issue deploying our web tier with DSC getting locked up on a certain reboot thinking it needed a reboot to finish but coming up in the same state after a reboot in an endless cycle. We also hit issues with our deployment of the git client on those machines.
  • We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.
  • We learned that if you move the drives from a RAID array from an R610 to an R630 and don’t catch the PXE boot prompt, the server will happily boot all the way into the OS.
  • We learned the good and the bad of the Dell FX2 IOA architecture and how they are self-contained switches.
  • We learned the CMC (management) ports on the FX2 chassis are effectively a switch. We knew they were suitable for daisy chaining purposes. However, we promptly forgot this, plugged them both in for redundancy and created a switching loop that reset Spanning Tree on our management network. Oops.
  • We learned the one guy on twitter who was OCD about the one upside down box was right. It was a pain to flip that web server over after opening it upside down and removing some critical box supports.
  • We didn’t mention this was a charge-only cable. Wow, that one riled twitter up. We appreciate the #infosec concern though!
  • We drastically underestimated how much twitter loves naked servers. It’s okay, we do too.
  • We learned that Dell MD1400 (13g and 12Gb/s) DAS (direct attached storage) arrays do not support hooking into their 12g servers like our R620 backup server. We’re working with them on resolving this issue.
  • We learned Dell hardware diagnostics don’t even check the power supply, even when the server has an orange light on the front complaining about it.
  • We learned that Blizzards are cold, the wind is colder, and sleep is optional.

The Payoff

Here’s what the average render time for question pages looks like, if you look really closely you can guess when the upgrade happened: Question page render times The decrease on question render times (from approx 30-35ms to 10-15ms) is only part of the fun. The next post in this series will detail many of the other drastic performance increases we’ve seen as the result of our upgrades. Stay tuned for a lot of real world payoffs we’ll share in the coming weeks.

Does all this sound like fun?

To us, it is fun. If you feel the same way, come do it with us. We are specifically looking for sysadmins preferably with data center experience to come help out in New York. We are currently hiring 2 positions:

If you’re curious at all, please ask us questions here, Twitter, or wherever you’re most comfortable. Really. We love Q&A.

Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.

How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.

Observability is the Foundation

Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:

  • Questions operators have about their systems can be quantifiably answered with minimal effort
  • Operators have rich mental models of how their systems function

When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.

Decision Making and Incident Preparedness

Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.

OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.

Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.

In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.

You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.

Other Benefits

Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.

Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.

Convinced? 5 Steps to Achieving Good Observability:

In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.

  1. Instrument your systems by publishing metrics and events
  2. Gather those metrics and events in a queryable data store(s)
  3. Make that data readily accessible
  4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
  5. Establish the resources to drill down into abnormal or out of bounds behavior

Each of these steps largely depends on the previous step to be successful.

1. Instrument your Systems

Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:

  1. Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
  2. Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.

Good Metrics also tend to have these properties:

  • High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
  • Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
  • Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.

It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.

2. Gather those metrics in a queryable data store(s)

This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:

  • Building dashboards
  • Enabling capacity planning
  • Allowing operators to explore the data and learn
  • Allowing people to invent cool stuff you didn’t anticipate

As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:

  • Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
  • Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views

3. Make that data Readily Accessible

If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:

  • A fast responsive UI to allow for operators to drill down and explore easily
  • Enables operators to create their own dashboards and graphs
  • Highlight problems

4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior

Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.

Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.

Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.

5. Establish the resources to drill down into abnormal or out of bounds behavior

The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:

  1. Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
  2. Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
  3. Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)

Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.

Use the science, Luke

If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.

If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

  • Monitoring allows you to send alerts if certain conditions are met,
  • It allows you to visualize trends in data,
  • Provides a method of assurance to the customer that their consumed services are guarded,
  • Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
A Large Network Operations Center

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.


One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.


Xzibit meme targeting monitoring systems

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

It has been my experience that many people simply buy a wireless access point, plop it down squarely next to their home cable/dsl modem, and assume that’s all they have to do to maximize their WiFi experience. Oh, were it to be so simple! I’d like to take a few minutes of your time today to cover some of the basics of what WiFi is, what it is and is not capable of, and how you as a SysAdmin or a home user can do a bit of detective work to help ensure your WiFi experience is less prone to issue.

RF Basics

Let’s take a moment and talk about Radio-Frequency Radiation. RF is a form of non-ionizing radiation where waves of energy radiate from a source and follow a predictable pattern based on the transmitter power and antenna. Radio waves are measured based on the size of the wave, and how frequently the wave oscillates. The frequency is measured in Hertz (Hz), or cycles per second.

Animated gif visualizing Hertz.  Image found on wikipedia, courtesy of Superborsuk.

Wavelength is the distance the radiation travels before it completes a single cycle. As we are mentioning travelling, we need to know the speed, right? This, my friends, is the speed of light.

C = f * λ , which translates to: Speed of Light = Frequency * Wavelength OR, if you're lazy, 300/Frequency in megahertz. Light travels approximately 300 million meters per second, we can drop a whole bunch of zeros from the equation and still be reasonably accurate.

WiFi signals operate on 2.4 gHz (2.4 billion cycles per second), and that means that one full wave travels around 12-13 centimeters before the waveform returns to its starting position relative to the axis in the graph. 802.11a and 802.11n operate on the 5gHz range, which would put the signal wavelength at 6 centimeters.

OK, but, why should I care about this when all I want to do is surf porn and play online games? The answer lies in the fact that if your antenna is not properly suited for these measurements, it won’t work that well. The antennas you get from your access point vendor are “suitable” but far from ideal.

Not many people realize it, but there is an aftermarket for antennas for access points. When people/companies buy commercial grade access points, they usually don’t include any antennas, as it’s assumed you’re going to get the proper antenna for your application.

So, what types of antennas are there and what are the differences?

Omnidirectional – These are the antennas that people are most familiar with. They will usually be oriented vertically, and radiate their signal on the horizontal plane in all 360 degrees. See below radiation chart which does a good job of visualizing how the energy travels out of a veritcal omni antenna.

Vertical omnidirectional antenna radiation pattern.  Image from wikipedia, courtesy of user LP.

Directional/Yagi – Directional antennas are designed to send a signal straight to a specific spot with pinpoint accuracy. If you’re trying to setup a WiFi link between your house and a neighbor down the street, you’d need a directional antenna. The Pringles Cantenna is an example of a homemade directional antenna. Commercial antennas more closely resemble old TV antennas that everyone seemed to have on their house back in the 20th century. The below radiation pattern does look a bit weird, but understand that the directional beam is designed to be highly selective of signals based on its relative orientation versus the target signal. This allows a directional antenna to receive and send to stations much further away than an omnidirectional antenna, which sends RF energy in all directions.

directional antenna radiation pattern.  Image is unattributed; if you own it contact me to remove or get credit.

patch – Patch antennas are normally flat antennas that are designed to radiate in a forward direction extremely well, with the signal attenuating sharply at the periphery. The radiation pattern below does have some similarities to the directional/yagi radiation pattern, but its lobe is more rounded in the forward direction. The patch antenna type is a good choice when you want to direct most of your energy in a particular direction but don’t necessarily want the pinpoint accuracy of a yagi.

patch antenna radiation pattern.  Image is unattributed; if you own it contact me to remove or get credit.

What blocks WiFi?

WiFi, operating in the 2.4ghz and 5ghz ranges, propogates in “line of sight.” Due to the short wavelength, the energy dissipates quicker if it is not channeled into a high-gain directional antenna. Consumer grade access points come with omnidirectional rubber duck antennas, which people usually orient vertically. If you look at the above radiation pattern, you’ll see that there is a void of energy directly above omnidirectional antennas when they are oriented horizontally.

All matter will attenuate RF energy to some extent as it passes along. The question on many people’s minds is what are the worst places you can install a wireless router or access point? Believe it or not, most materials in the home are not capable of attenuating your WiFi signal to a noticeable degree. In order for WiFi signals to be blocked effectively, they need to move through several layers of dense material in order to shed the energy required to become unusable. Some antenna manufacturers will quote how well the radio waves will propogate from a given antenna, as shown here for one of Cisco’s branded antennas:

The density of the materials used in a building’s construction determines the number of walls the signal must pass through and still maintain adequate coverage. Consider the following before choosing the location to install your antenna:
  • Paper and vinyl walls have very little affect on signal penetration.
  • Solid and pre-cast concrete walls limit signal penetration to one or two walls without degrading coverage.
  • Concrete and wood block walls limit signal penetration to three or four walls.
  • A signal can penetrate five or six walls constructed of drywall or wood.
  • A thick metal wall causes signals to reflect off, causing poor penetration.
  • A wire mesh spaced between 1 and 1 1/2 in. (2.5 and 3.8 cm) acts as a harmonic reflector that blocks a 2.4-Ghz radio signal. (NOTE: as a commenter below further explains, this type of mesh is common in plaster walls from the 1940s as well as in stucco applications.)

I once ran into an issue with a friend who had his wireless router installed in the basement, next to his cablemodem. He was having sporadic connectivity issues in a second floor room and asked me to come help diagnose. Sure enough, his room was directly above the wireless router, two residential floors below, and given that traverse and the location of his room in relation to the radiation pattern, there wasn’t enough RF energy propogating up into that location. The short-term answer for the problem was to orient his access point antennas horizontally, so that the radiation pattern is then set on its side, covering a wider swath of his house.

In a blog post to come, I will show you some methods you can use to help properly locate your access point and also help you decide which frequency your access point should operate on. Stay tuned, and as always your comments,criticisms and suggestions are always welcome!

In the never ending quest for better performance and response time, perhaps consistency is underrated. In web operations the desire is to make page loads go fast, because performance is a feature, or more simply — people like responsive web pages.

Probably the most common and simplest way to look at response time is to look at the average response time for the primary web request in a page load. Page loads also need to be looked at from the perspective of how they render in a browser (Which is how Google looks at it). This all makes sense, but as you push your load times faster and faster, the savings start to get smaller and smaller. For instance you start reducing the average time of certain requests from something like 1 second to 500 MS, then 500 to 100, then 100 to 80, 50, 30, 15, 10. Eventually these push to limits where you are less likely to feel the difference. It doesn’t mean it is time to stop, but I think in general it makes sense to shift your attention elsewhere for performance gains.

When this happens, a next logical step is often to go for a Content Delivery Network. The CDN can help reduce load times for people who are geographically far from the main servers. We did this back in May for our static content. Then what?

I think a next logical step is to start to shift attention to the consistency of your web server’s response time. By consistency I mean addressing those responses that take a lot longer than your average request time because of the server’s response time. In other words, fixing the outliers. To address these, is quite frankly, a pain. Why?

  • You need a decent sized sample of the response time of every request
  • At hundreds of requests a second, it is hard to pin down the cause of these outliers. It can be code, DB behavior, network blips, etc.

The best way I have come up with to measure this is to pull the response time from our web logs as our load balancer measures it from:

Server response time (HTTP mode only). It’s the time elapsed between the moment the TCP connection was established to the server and the moment the server sent its complete response headers. It purely shows its request processing time, without the network overhead due to the data transmission…

To be honest, I am not sure if this is the best way, but it seems like a reasonable start. I then filtered a sample of log requests so that I was only looking at authenticated users loading a question on stackoverflow.com that got a 200 response code from the web server (to ease in taking crawlers out of the mix). Then to express the outliers I used a box-and-whisker plot. If you have been wondering just what that image on the right is — this is what you are looking at. You can go read more about it, but in short, the dots are the outliers. When you look at this as a histogram these outliers will be the long tail (which in this case pushes the average to the right of the median).

I have been giving Marc Gravell some data and we were able to pin down a lot of these as collateral damage to a particularly demanding background thread that runs on each server. He is reworking this to help eliminate a lot of these.

I think keeping an eye on these outliers or the long tail of response is going to be tricky, but I also think it plays an important role in the quest for better performance.

The Three Perspectives

In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:

  • Micro: “Ground Level”
  • Meso: “Day to Day”
  • Macro: “Seasonal”

These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.

From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.

I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.

Ground Level Monitoring

“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:

  • Perfmon
  • Sar
  • Wireshark / TCPDump
  • Web Logs (or other detailed logs)
  • SQL Server Profiler

However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.

In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.

The Attributes of Micro Data

The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:

  • Different sets of tools to process the data are needed
  • Generally samples of high resolution data and not complete sets are used
  • To correlate all of this, you will need to do some work because it will be different for every environment

Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.

One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).

Case Study: Web Logs and Perfmon Data

Data Analysis Platforms

Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.

Getting the Data

For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.

Exploring the Data with R

I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):

Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.

The Difference?

If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.

What is Next?

I don’t have a high resolution system deployed yet. So I need work on and think about:

  • Scheduling Perfmon
  • How to get the most representative samples
  • Scheduling TCPDump and getting that data into R
  • Scheduling Sar on the Linux boxes and getting that into R
  • Automating parts of the analysis

I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.

In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.

Transfer rates and the number of packets you send are measured in units of a certain quantity of data per units of time. The unit of time that everyone is used to is the second. The standard quantity of data that is used in the networking field is bits and the standard time unit is seconds. So for example, the standard network interface these days is 1 Gigabit per second. So the quantity of data is a Gigabit, and the unit of time is a second. We call this the transfer rate. The key thing to remember is that this is a fixed ratio of data over time. Because of this, you can divide the ratio by any number you want to (Ignoring the complexities of the discrete properties of Ethernet frequencies, system clocking, etc). So, 500 Mbit over a half second is the same fixed ratio as 1 Gigabit per second.

The thing is though, in computing, a second is a really, really, really long time. This is important, because when we choose what unit of time to express this in, what we are doing is graph smoothing (It is sort of, although not really, like taking an average).

For example, we could transfer 900 Mbit in half of a second and another 100 Mbit for the other half of that second. How much data was transferred during that second? The answer is 1 Gbit. If we transfer 500 Mbit per half second and another 500 Mbit per the other half second — this is also 1 Gbit per second:. This effect is illustrated in these Megabits per half second graphs:

These two are clearly not the same thing, but when you express them as the amount of data transfered over a second they are. This is important because a 1 Gbit per second interface is also a 500 Mbit per half second interface — and a 500 Mbit per half second interface can’t transfer 900 Mbits per half second (I am ignoring any buffering effects, but in practice we have found this to be essentially true).

This effect is made even worse by most monitoring tools because most take samples every 5 minutes. So what you are really seeing is the transfer rate per 5 minutes converted to a per second rate. This sort of thing is why people say data can lie.

Why Should you Care?

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

We have bonded these interfaces using Intel Load Balancing (ALB/RLB) and for the most part our discards have gone away. We did this on all but one of our web servers for a while and found that the one that didn’t have the bonded interface had discards climbing while the others did not.

A second is a long time — be wary of trusting it too much to measure things.

Just about every introductory class for algorithms teaches you three primary ways to look at algorithm performance:

  • Worse Case Scenario
  • Average Case Scenario
  • Best Case Scenario (This is always followed up by why this isn’t very useful, but they mention it anyways).

You can use this same sort of analysis when thinking about your web performance. I have found that a histogram captures all 3 of these pretty nicely since it is a distribution. The X axis is the response time in milliseconds of a request, and the Y axis is the number of hits that had that response time . Our HAProxy web logs capture the response time of each request from the perspective of the load balancer. This is a nice perspective because it includes pretty much the full stack of a web request that is directly under our control.

We have started to insert our web logs into SQL server. Raw SQL is hard to beat for deep analysis, but I have also started to build a web front end which is particularly useful for generating graphs easily. With a web interface it is easy to filter on certain criteria and you can easily see the distribution for a particular page or a certain client like Googlebot:

(The X axis is the server response in MS, and the Y axis is number of hits that fell into that response bracket)

A crawler will also be a sort of worst case response time perspective on another level because because crawlers cause more cache misses. Clients will generally perform much differently. For example looking at response time for user agents with “Chrome” in them the response time has a very different shape:

In general, I always find it is best to think about how you are viewing your data, and if it is the best way to summarize what you really care about. Average is useful, but it just isn’t the complete picture.

We recently changed the NICs in our web tier and primary database servers from Broadcom to Intel based NICs based on some … issues we had been having. After we put them in they worked reasonably well, but we knew that they could be faster and push more data. When I started to dig into just what we could do to tweak the pleathora of settings for the new NICs I found a few settings that would probably help a little bit, and one technology that could help us out tremendously.

Changes at a glance

  • Turned on Intel I/OAT
  • Adjusted Send and Recieve buffers to 2048 (max allowed)
  • Turned off interrupt moderation
  • Increased the Receive Side Scaling Queues from 1 to 4

The long version

The first and biggest change is that we turned on Intel’s I/OAT technology. It’s a collection of different techniques that work together to improve the performance of your host networking, as defined on Intel’s website:

  • Intel® QuickData Technology — enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput.
  • Direct Cache Access (DCA) — allows a capable I/O device, such as a network controller, to place data directly into CPU cache, reducing cache misses and improving application response times.
  • Extended Message Signaled Interrupts (MSI-X) – distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance.
  • Receive Side Coalescing (RSC) — aggregates packets from the same TCP/IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing.
  • Low Latency Interrupts — tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency.

The catch is you need to be running a full Intel hardware stack. Your CPU, Motherboard, BIOS, NIC and OS all need to be compatible with the technology to be able to use it. Once you have the right stack in place, you might have to turn on a BIOS option, but that’s it no tweaking or poking to make it just right, it’s just right out of the box. Turning it on was as simple as flipping a BIOS setting, aptly named “Intel I/OAT.”

Performance tuning options

The performance options are all exposed via the PROSet utility making it nice and easy to change, no need to go digging into the registry for some esoteric key that may or may not be there. For each of them there is a trade off you need to consider. Some of the options will increase host CPU, some will cause higher host memory usage. To find the right value for your systems you really need to evaluate your overall situation and see if the trade offs are worth it.

Send and Receive Buffers

The Send and Receive buffers where set to the maximum allowed value of 2048. The trade off here is that you will consume more host memory. For us this is not a big deal since we have a lot of RAM on our boxes. Also, we had been seeing a good deal of Zero-window TCP packets when investigating our network so we needed to increase the buffer anyway.

Interrupt Moderation

The Interrupt Moderation feature was disabled. This feature allows you to have the NIC throttle the number of interrupts to the CPU which will limit the number of CPU cycles used by the NIC interrupts. Turning this off will increase your CPU usage, but it will also prevent packets from sitting there waiting for an interrupt to be proccessed. The increased cpu is a pain point for us right now (we are working on fixing that) but I believe it’s worth it.

Receive Side Scaling Queues

Receive Side Scaling is a technology that allows you to process a TCP connection across multiple cores. This allows for more efficient cache and processor usage when you TCP connection is not tied to a single core. When you are using this feature it will only use real physical cores to process TCP connections, you are not able to use this with hyper thread cores.

Receive Side Scaling Queues are essentially buffer space that is used between the NIC and the CPU when you are using Receive Side Scaling. This is another setting that has a trade off between host CPU and performance. I opted for the trade off, and increased the queues from 1 to 4 queues.

Additional notes

  • Since I/OAT needs BIOS support and the BIOS on our web tier was woefully out of date anyway I updated to the latest BIOS for those machines
  • I updated the Intel PROSet utility to verion 16.1 from 16.0

We are less than twenty four hours into using these new setting, but everything looks much much better through our peak traffic today. So far we are very happy with the results of these changes.