A few weeks ago we upgraded a lot of the core infrastructure in our New York (okay, it’s really in New Jersey now – but don’t tell anyone) data center. We love being open with everything we do (including infrastructure), and really consider it one of the best job perks we have. So here’s how and why we upgrade a data center. First, take a moment to look at what Stack Overflow started as. It’s 5 years later and hardware has come a long way.

Why?

Up until 2 months ago, we hadn’t replaced any servers since upgrading from the original Stack Overflow web stack. There just hasn’t been a need since we first moved to the New York data center (Oct 23rd, 2010 – over 4 years ago).  We’re always reorganizing, tuning, checking allocations, and generally optimizing code and infrastructure wherever we can. We mostly do this for page load performance; the lower CPU and memory usage on the web tier is usually a (welcomed) side-effect.

So what happened? We had a meetup. All of the Stack Exchange engineering staff got together at our Denver office in October last year and we made some decisions. One of those decisions was what to do about infrastructure hardware from a lifecycle and financial standpoint. We decided that from here on out: hardware is good for approximately 4 years. After that we will: retire it, replace it, or make an exception and extend the warranty on it. This lets us simplify a great many things from a management perspective, for example: we limit ourselves to 2 generations of servers at any given time and we aren’t in the warranty renewal business except for exceptions. We can order all hardware up front with the simple goal of 4 years of life and with a 4 year warranty.

Why 4 years? It seems pretty arbitrary. Spoiler alert: it is. We were running on 4 year old hardware at the time and it worked out pretty well so far. Seriously, that’s it: do what works for you. Most companies depreciate hardware across 3 years, making questions like “what do we do with the old servers?” much easier. For those unfamiliar, depreciated hardware effectively means “off the books.” We could re-purpose it outside production, donate it, let employees go nuts, etc. If you haven’t heard, we raised a little money recently. While the final amounts weren’t decided when we were at the company meetup in Denver, we did know that we wanted to make 2015 an investment year and beef up hardware for the next 4.

Over the next 2 months, we evaluated what was over 4 years old and what was getting close. It turns out almost all of our Dell 11th generation hardware (including the web tier) fits these criteria – so it made a lot of sense to replace the entire generation and eliminate a slew of management-specific issues with it. Managing just 12th and 13th generation hardware and software makes life a lot easier – and the 12th generation hardware will be mostly software upgradable to near equivalency to 13th gen around April 2015.

What Got Love

In those 2 months, we realized we were running on a lot of old servers (most of them from May 2010):

  • Web Tier (11 servers)
  • Redis Servers (2 servers)
  • Second SQL Cluster (3 servers – 1 in Oregon)
  • File Server
  • Utility Server
  • VM Servers (5 servers)
  • Tag Engine Servers (2 servers)
  • SQL Log Database

We also could use some more space, so let’s add on:

  • An additional SAN
  • An additional DAS for the backup server

That’s a lot of servers getting replaced. How many? This many: Greg Bray, and a lot of old servers.

The Upgrade

I know what you’re thinking: “Nick, how do you go about making such a fancy pile of servers?” I’m glad you asked. Here’s how a Stack Exchange infrastructure upgrade happens in the live data center. We chose not to failover for this upgrade; instead we used multiple points of redundancy in the live data center to upgrade it while all traffic was flowing from there.

Day -3 (Thursday, Jan 22nd): Our upgrade plan was finished (this took about 1.5 days total), including everything we could think of. We had limited time on-site, so to make the best of that we itemized and planned all the upgrades in advance (most of them successfully, read on). You can find a read the full upgrade plan here.

Day 0 (Sunday, Jan 25th): The on-site sysadmins for this upgrade were George Beech, Greg Bray, and Nick Craver (note: several remote sysadmins were heavily involved in this upgrade as well: Geoff Dalgas online from Corvallis, OR, Shane Madden, online from Denver, CO, and Tom Limoncelli who helped a ton with the planning online from New Jersey). Shortly before flying in we got some unsettling news about the weather. We packed our snow gear and headed to New York.

Day 1 (Monday, Jan 26th): While our office is in lower Manhattan, the data center is now located in Jersey City across the Hudson river: Office to the Data Center We knew there was a lot to get done in the time we had allotted in New York, weather or not. The thought was that if we skipped Monday we likely couldn’t get back to the data center Tuesday if the PATH (mass transit to New Jersey) shut down. This did end up happening. The team decision was: go time. We got overnight gear then headed to the data center. Here’s what was there waiting to be installed:

Web, Redis, and Service serversNew 10Gb network gearFX2s Blade Chassis for VMs

Yeah, we were pretty excited too. Before we got started with the server upgrade though, we first had to fix a critical issue with the redis servers supporting the launching-in-24-hours Targeted Job Ads. These machines were originally for Cassandra (we broke that data store), then Elasticsearch (broke that too), and eventually redis. Curious? Jason Punyon and Kevin Montrose have an excellent blog series on Providence, you can find Punyon’s post on what broke with each data store here.

The data drives we ordered for these then-redundant systems were the Samsung 840 Pro drives which turned out to have a critical firmware bug. This was causing our server-to-server copies across dual 10Gb network connections to top out around 12MB/s (ouch). Given the hundreds of gigs of memory in these redis instances, that doesn’t really work. So we needed to upgrade the firmware on these drives to restore performance. This needed to be online, letting the RAID 10 arrays rebuild as we went. Since you can’t really upgrade firmware over most USB interfaces, we tore apart this poor, poor little desktop to do our bidding:

Once that was kicked off, it ran in parallel with other work (since RAID 10s with data take tens of minutes to rebuild, even with SSDs). The end result was much improved 100-200MB/s file copies (we’ll see what new bottleneck we’re hitting soon – still lots of tuning to do). Now the fun begins. In Rack C (we have high respect for our racks, they get title casing), we wanted to move from the existing SFP+ 10Gb connectivity combined with 1Gb uplinks for everything else to a single dual 10Gb BASE-T (RJ45 connector) copper solution. This is for a few reasons: The SFP+ cabling we use is called twinaxial which is harder to work with in cable arms, has unpredictable girth when ordered, and can’t easily be gotten natively in the network daughter cards for these Dell servers. The SFP+ FEXes also don’t allow us to connect any 1Gb BASE-T items that we may have (though that doesn’t apply in this rack, it does when making it a standard across all racks like with our load balancers). So here’s what we started with in Rack C:

What we want to end up with is:

The plan was to simplify network config, cabling, overall variety, and save 4U in the process. Here’s what the top of the rack looked like when we started: …and the middle (cable management covers already off):

Let’s get started. First, we wanted the KVMs online while working so we, ummm, “temporarily relocated” them: Now that those are out of the way, it’s time to drop the existing SFP+ FEXes down as low as we could to install the new 10Gb BASE-T FEXes in their final home up top: The nature of how the Nexus Fabric Extenders work allows us to allocate between 1 and 8 uplinks to each FEX. This means we can unplug 4 ports from each FEX without any network interruption, take the 4 we find dead in the VPC (virtual port channel) out of the VPC and assign them to the new FEX. So we go from 8/0 to 4/4 to 0/8 overall as we move from old to new through the upgrade. Here’s the middle step of that process: With the new network in place, we can start replacing some servers. We yanked several old servers already, one we virtualized and 2 we didn’t need anymore. Combine this with evacuating our NY-VM01 & NY-VM02 hosts and we’ve made 5U of space through the rack. On top of NY-VM01&02 was 1 of the 1Gb FEXes and 1U of cable management. Luckily for us, everything is plugged into both FEXes and we could rip one out early. This means we could spin up the new VM infrastructure faster than we had planned. Yep, we’re already changing THE PLAN™. That’s how it goes. What are we replacing those aging VM servers with? I’m glad you asked. These bad boys:

There are 2 of these Dell PowerEdge FX2s Blade Chassis each with 2 FC630 blades. Each blade has dual Intel E5-2698v3 18-core processors and 768GB of RAM (and that’s only half capacity). Each chassis has 80Gbps of uplink capacity as well via the dual 4x 10Gb IOA modules. Here they are installed:

The split with 2 half-full chassis give us 2 things: capacity to expand by double, and avoiding any single points of failure with the VM hosts. That was easy, right? Well what we didn’t plan on was the network portion of the day, it turns out those IO Aggregators in the back are pretty much full switches with 4 external 10Gbps ports and 8 internal 10Gbps (2 per blade) ports each. Once we figured out what they could and couldn’t do, we got the bonding in place and the new hosts spun up.

It’s important to note here it wasn’t any of the guys in the data center spinning up this VM architecture after the network was live. We’re setup so that Shane Madden was able to do all this remotely. Once he had the new NY-VM01 & 02 online (now blades), we migrated all VMs over to those 2 hosts and were able to rip out the old NY-VM03-05 servers to make more room. As we ripped things out, Shane was able to spin up the last 2 blades and bring our new beasts fully online. The net result of this upgrade was substantially more CPU and memory (from 528GB to 3,072GB overall) as well as network connectivity. The old hosts each had 4x 1Gb (trunk) for most access and 2x 10Gb for iSCSI access to the SAN. The new blade hosts each have 20Gb of trunk access to all networks to split as they need.

But we’re not done yet. Here’s the new EqualLogic PS6210 SAN that went in below (that’s NY-LOGSQL01 further below going in as well):

VM Servers, SAN, and NY-LOGSQL01 Our old SAN was a PS6200 with 24x 900GB 10k drives and SFP+ only. This is a newer 10Gb BASE-T 24x 1.2TB 10k version with more speed, more space, and the ability to go active/active with the existing SAN. Along the the SAN we also installed this new NY-LOGSQL01 server (replacing an aging Dell R510 never designed to be a SQL server – it was purchased as a NAS):

The additional space freed by the other VM hosts let us install a new file and utility server:

Of note here: the NY-UTIL02 utility server has a lot of drive bays so we could install 8x Samsung 840 Pros in a RAID 0 in order to restore and test the SQL backups we make every night. It’s RAID 0 for space because all of the data is literally loaded from scratch nightly – there’s nothing to lose. An important lesson we learned last year was that the 840 Pros do not have capacitors in there and power loss will cause data loss if they’re active since they have a bit of DIMM for write cache on board. Given this info – we opted to stick some Intel S3700 800GB drives we had from the production SQL server upgrades into our NY-DEVSQL01 box and move the less resilient 840s to this restore server where it really doesn’t matter.

Okay, let’s snap back to blizzard reality. At this point mass transit had shut down and all hotels in (blizzard) walking distance were booked solid. Though we started checking accommodations as soon as we arrived on site, we had no luck finding any hotels. Though the blizzard did far less than predicted, it was still stout enough to shut everything down. So, we decided to go as late as we could and get ahead of schedule. To be clear: this was the decision of the guys on site, not management. At Stack Exchange employees are trusted to get things done, however they best perceive how to do that. It’s something we really love about this job.

If life hands you lemons, ignore those silly lemons and go install shiny new hardware instead.

This is where we have to give a shout out to our data center QTS. These guys had the office manager help us find any hotel we could, set out extra cots for us to crash on, and even ordered extra pizza and drinks so we didn’t go starving. This was all without asking – they are always fantastic and we’d recommend them to anyone looking for hosting in a heartbeat.

After getting all the VMs spun up, the SAN configured, and some additional wiring ripped out, we ended around 9:30am Tuesday morning when mass transit was spinning back up. To wrap up the long night, this was the near-heart attack we ended on, a machine locking up at: BIOS Lockup Turns out a power supply was just too awesome and needed replacing. The BIOS did successfully upgrade with the defective power supply removed and we got a replacement in before the week was done. Note: we ordered a new one rather than RMA the old one (which we did later). We keep a spare power supply for each wattage level in the data center, and try to use as few different levels as possible.

Day 2 (Tuesday, Jan 27th): We got some sleep, got some food, and arrived on site around 8pm. Starting the web tier (a rolling build out) was kicked off first:

A stack of web serversA line of web serversSame line!Inside a web server

While we rotated 3 servers at a time out for rebuilds on the new hardware, we also upgraded some existing R620 servers from 4x 1Gb network daughter cards to 2x 10Gb + 2x 1Gb NDCs. Here’s what that looks like for NY-SERVICE03:

A line of web serversSame line!Inside a web server

The web tier rebuilding gave us a chance to clean up some cabling. Remember those 2 SFP+ FEXes? They’re almost empty: The last 2 items were the old SAN and that aging R510 NAS/SQL server. This is where the first major hiccup in our plan occurred. We planned to install a 3rd PCIe card in the backup server pictured here: We knew it was a Dell R620 10 bay chassis that has 3 half-height PCIe cards. We knew it had a SAS controller for the existing DAS and a PCIe card for the SFP+ 10Gb connections it has (it’s in the network rack with the cores in which all 96 ports are 10Gb SFP+). Oh hey look at that, it’s hooked to a tape drive which required another SAS controller we forgot aboutCrap. Okay, these things happen. New plan.

We had extra 10Gb network daughter cards (NDCs) on hand, so we decided to upgrade the NDC in the backup server, remove the SFP+ PCIe card, and replace it with the new 12Gb SAS controller. We also forgot to bring the half-height mounting bracket for the new card and had to get creative with some metal snips (edit: turns out it never came with one – we feel slightly less dumb about this now). So how do we plug that new 10Gb BASE-T card into the network core? We can’t. At least not at 10Gb. Those 2 last SFP+ items in Rack C also need a home – so we decided to make a trade. The whole backup setup (including  new MD1400 DAS) just love their new Rack C home:

Then we could finally remove those SFP+ FEXes, bring those KVMs back to sanity, and clean things up in Rack C:

Those pesky hanging KVMsTop of Rack CMiddle of Rack C

See? There was a plan all along. The last item to go in Rack C for the day is NY-GIT02, our new Gitlab and TeamCity server:

Signatures from the New York devsRacked and ready to go

Note: we used to run TeamCity on Windows on NY-WEB11. Geoff Dalgas threw out the idea during the upgrade of moving it to hardware: the NY-GIT02 box. Because they are such intertwined dependencies (for which both have an offsite backup), combining them actually made sense. It gave TeamCity more power, even faster disk access (it does a lot of XML file…stuff), and made the web tier more homogenous all at the same time. It also made the downtime of NY-WEB11 (which was imminent) have far less impact. This made lots of sense, so we changed THE PLAN™ and went with it. More specifically, Dalgas went with it and set it all up, remotely from Oregon. While this is happening, Greg was fighting with a DSC install hang regarding git on our web tier: Greg losing to DSC Wow that’s a lot of red, I wonder who’s winning. And that’s Dalgas in a hangout on my laptop, hi Dalgas! Since the web tier builds were a relatively new process fighting us, we took the time to address some of the recent cabling changes. The KVMs were installed hastily not long before this because we knew a re-cable was coming. In Rack A for example we moved the top 10Gb FEX up a U to expand the cable management to 2U and added 1U of management space between the KVMs. Here’s that process:

A messy starting KVMRemoving the cable management to make roomAhhhh room!That's better, all done.

Since we had to re-cable from the 1Gb middle FEXes in Rack A & B (all 4 being removed) to the 10Gb Top-of-Rack FEXes, we moved a few things around. The CloudFlare load balancers down below the web tier at the bottom moved up to spots freed by the recently virtualized DNS servers to join the other 2 public load balancers. The removal of the 1Gb FEXes as part of our all-10Gb overhaul meant that the middle of Racks A & B had much more space available, here’s the before and after:

Web tier below a 1Gb FEXLook at all that space!

After 2 batches of web servers, cable cleanup, and network gear removal, we called it quits around 8:30am to go grab some rest. Things were moving well and we only had half the web tier, cabling, and a few other servers left to replace.

Day 3 (Wednesday, Jan 28th): We were back in the data center just before 5pm, set up and ready to go. The last non-web servers to be replaced were the redis and “service” (tag engine, elasticsearch indexing, etc.) boxes:

A look inside redisNY-REDIS01 and NY-SERVICE05 racked and ready for an OS

We have 3 tag engine boxes (purely for reload stalls and optimal concurrency, not load) and 2 redis servers in the New York data center. One of the tag engine boxes was a more-recent R620, (this one got the 10Gb upgrade earlier) and wasn’t replaced. That left NY-SERVICE04, NY-SERVICE05, NY-REDIS01 and NY-REDIS02. On the service boxes the process was pretty easy, though we did learn something interesting: if you put both of the drives from the RAID 10 OS array in an R610 into the new R630…it boots all the way into Windows 2012 without any issues. This threw us for a moment because we didn’t remember building it in the last 3 minutes. Rebuild is simple: lay down Windows 2012 R2 via our image + updates + DSC, then install the jobs they do. StackServer (from a sysadmin standpoint) is simply a windows service – and our TeamCity build handles the install and such, it’s literally just a parameter flag. These boxes also run a small IIS instance for internal services but that’s also a simple build out. The last task they do is host a DFS share, which we wanted to trim down and simplify the topology of, so we left them disabled as DFS targets and tackled that the following week – we had NY-SERVICE03 in rotation for the shares and could do such work entirely remotely. For redis we always have a slave chain happening, it looks like this: This means we can do an upgrade/failover/upgrade without interrupting service at all. After all those buildouts, here’s the super fancy new web tier installed:

To get an idea of the scale of hardware difference, the old web tier was Dell R610s with dual Intel E5640 processors and 48GB of RAM (upgraded over the years). The new web tier has dual Intel 2687W v3 processors and 64GB of DDR4 memory. We re-used the same dual Intel 320 300GB SSDs for the OS RAID 1. If you’re curious about specs on all this hardware – the next post we’ll do is a detailed writeup of our current infrastructure including exact specs.

Day 4 (Thursday, Jan 29th): I picked a fight with the cluster rack, D. Much of the day was spent giving the cluster rack a makeover now that we had most of the cables we needed in. When it was first racked, the pieces we needed hadn’t arrived by go time. It turns out we were still short a few cat and power cables as you’ll see in the photos, but we were able to get 98% of the way there.

It took a while to whip this rack into shape because we added cable arms where they were missing, replaced most of the cabling, and are fairly particular about the way we do things. For instance: how do you know things are plugged into the right port and where the other end of the cable goes? Labels. Lots and lots of labels. We label both ends of every cable and every server on both sides. It adds a bit of time now, but it saves both time and mistakes later.

Cable labels!Web servers without labelsWeb servers with labels!Web server rear labels

Here’s what the racks ended up looking like when we ran out of time this trip:

It’s not perfect since we ran out of several cables of the proper color and length. We have ordered those and George will be tidying the last few bits up.

I know what you’re thinking. We don’t think that’s enough server eye-candy either.

Here’s the full album of our move.

And here’s the #SnowOps twitter stream which has a bit more.

What Went Wrong

  • We’d be downright lying to say everything went smoothly. Hardware upgrades of this magnitude never do. Expect it. Plan for it. Allow time for it.
  • Remember when we upgraded to those new database servers in 2010 and the performance wasn’t what we expected? Yeah, that. There is a bug we’re currently helping Dell track down in their 1.0.4/1.1.4 BIOS for these systems that seems to not respect whatever performance setting you have. With Windows, a custom performance profile disabling C-States to stay at max performance works. In CentOS 7, it does not – but disabling the Intel PState driver does. We have even ordered and just racked a minimal R630 to test and debug issues like this as well as test our deployment from bare metal to constantly improve our build automation. Whatever is at fault with these settings not being respected, our goal is to get that vendor to release an update addressing the issue so that others don’t get the same nasty surprise.
  • We ran into an issue deploying our web tier with DSC getting locked up on a certain reboot thinking it needed a reboot to finish but coming up in the same state after a reboot in an endless cycle. We also hit issues with our deployment of the git client on those machines.
  • We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.
  • We learned that if you move the drives from a RAID array from an R610 to an R630 and don’t catch the PXE boot prompt, the server will happily boot all the way into the OS.
  • We learned the good and the bad of the Dell FX2 IOA architecture and how they are self-contained switches.
  • We learned the CMC (management) ports on the FX2 chassis are effectively a switch. We knew they were suitable for daisy chaining purposes. However, we promptly forgot this, plugged them both in for redundancy and created a switching loop that reset Spanning Tree on our management network. Oops.
  • We learned the one guy on twitter who was OCD about the one upside down box was right. It was a pain to flip that web server over after opening it upside down and removing some critical box supports.
  • We didn’t mention this was a charge-only cable. Wow, that one riled twitter up. We appreciate the #infosec concern though!
  • We drastically underestimated how much twitter loves naked servers. It’s okay, we do too.
  • We learned that Dell MD1400 (13g and 12Gb/s) DAS (direct attached storage) arrays do not support hooking into their 12g servers like our R620 backup server. We’re working with them on resolving this issue.
  • We learned Dell hardware diagnostics don’t even check the power supply, even when the server has an orange light on the front complaining about it.
  • We learned that Blizzards are cold, the wind is colder, and sleep is optional.

The Payoff

Here’s what the average render time for question pages looks like, if you look really closely you can guess when the upgrade happened: Question page render times The decrease on question render times (from approx 30-35ms to 10-15ms) is only part of the fun. The next post in this series will detail many of the other drastic performance increases we’ve seen as the result of our upgrades. Stay tuned for a lot of real world payoffs we’ll share in the coming weeks.

Does all this sound like fun?

To us, it is fun. If you feel the same way, come do it with us. We are specifically looking for sysadmins preferably with data center experience to come help out in New York. We are currently hiring 2 positions:

If you’re curious at all, please ask us questions here, Twitter, or wherever you’re most comfortable. Really. We love Q&A.

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

  • More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
  • Licensing Costs
  • Owned vs Rented Assets
  • How many cloud “servers” or instances you would need vs real hardware
  • Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.

Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?

Why Use Virtualization?

Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.

This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:

  • Live migration (vMotion in VMWare terms)
  • Running multiple operating systems (i.e. Windows and Linux) on the same hardware
  • Easier to get full utilization of hardware resources by moving VMs around

These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?

The Joy of Commodity Hardware

As Wikipedia defines it:

Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”

Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.

So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.

Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.

Virtualization and Centralized Storage are Best Friends

With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.

Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.

Virtualized Clusters to the Rescue?

If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.

VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.

Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.

What Will We Go With?

Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.

I’ve recently been looking back on what we have written about our architecture in the past, and came to a stunning realization. That realization is that while we have many many different articles about what we have been doing there hasn’t been a good, solid overview of our architecture in a long time. In fact, the last really comprehensive write-up was done by Jeff before this blog even existed. And, boy I do have to say there has been quite a lot of change behind the scenes since then. So, my dear readers I’m going to take some time – and my next few blog posts – to give everyone an in depth look into how we have the Stack Exchange Network setup to serve between 12 and 14 Million page views per day.

How these posts will breakdown

Since we have obviously grown, and are offering more services to our users I’m going to break these posts out by each of the 4 major services we offer to our user base:

  • Core Q&A (this includes the API)
  • Careers
  • Chat
  • Community Blogs

Each one of these systems all work towards our goal of making the internet better, but they have different requirements and different challenges.

In this first post, I’ll be focusing on our core Q&A system, since that is after all our bread and butter.

Core Q&A

First, a high level overview of how everything is put together:

The Hardware

Our core hardware setup hasn’t changed all that much. Well, I should say the chassis haven’t changed that much. We’ve done a lot of work to upgrade the internals of the servers when needed to address performance issues as they came up, as well as handle issues that resulted from Stack Overflow being so big.

Web Tier

Of these 10 Servers, 3 are dedicated to Stack Overflow with an additional 3 servers serving Stack Overflow and the Stack Exchange Network. We have one server dedicated to Dev/QA – which also hosts meta.stackoverflow.com. Our Web Tier machines normally operate between 5 and 20% utilization. We have plenty of room to grow on these boxes.

  • 10 Dell R610 IIS web servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
    • 16 GB RAM
    • Windows Server 2008 R2
    • 2 drives
      • RAID 1
      • 2x Intel 320 300GB SSD (RAID 1)

DB Tier

We have two database server pairs. One pair is dedicated to running Stack Overflow, and the other runs the rest of the network. We run development against the secondary server of the non-stack overflow database pair. Both of our database pairs run at about 20% utilization, so once again we have room to grow here as well.

  • 2 Dell R710 database servers:
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 spindles
      • Mirrored Pair for OS
      • 6 disk RAID10 for databases
    • SQL Server 2008 R2 SP1
  • 2 Dell R710 database servers (Stack Overflow Dedicated):
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 drives
      • Mirrored Pair for OS
      • 6 drive RAID10 of Intel X25-E SSDs for Database
    • SQL Server 2008 R2 SP1

Caching Tier

We run redundant Redis servers for our caching tier.

  • 2 Dell R610 Redis servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz
    • 16 GB RAM
    • CentOS

Network Layer

We use HAProxy for our load balancing, and Cisco Switching.

  • 2 Dell R610 HAProxy servers:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 4 GB RAM
    • Ubuntu Server
  • 6 WS-C2960S-48TS-L Gigabit switches
    • FlexStack (two stacks, 4 switches and 2 switches)

Data Integrity

As with any system, making sure that your data is backed up and the backups are good is an integral part to your service offering. We backup our databases nightly and restore them to two different locations. One local to our NY data center for our devs to work against, and one remote in our OR data center.

Conclusion

Overall I believe that we are in a good place and have plenty of room to grow given our current setup. As always we will constantly be looking at our infrastructure and tweaking it to get the best performance possible and give our users the best experience possible.

Recently there was a question on the Programmers site asking “Why don’t all companies buy developers the best hardware?” This is actually a very interesting question and there was a good deal of discourse on the topic, as there always is when you get a group of professionals talking about the tools of their trade.

We here at Stack Exchange pride ourselves on not letting the technology get in the way of our devs. We want them to be able to do their job – which is writing code, and doing it well – with minimal hassle. Now, I have to take a brief moment to make a little bit of a note some of the things we do do not scale.

The philosophy that is maintained when getting a developer a new machine is a pretty simple one:

> If a dev is constantly struggling with their machine they aren’t getting work done, they aren’t happy, and they are producing bad code. All of these things are more expensive than a nice developer machine

As a sysadmin – and hence the guy in charge of getting the devs what they need you find out really quickly that they know what they want for the most part, and for those that you don’t ask if you give them a powerful machine with a lot of screen real estate they are generally very happy.

Since I mentioned earlier that there are some things we do here that aren’t all that scalable – in that they work for a company with 20 devs but not 100 i’ll split them up.

The Scalable

The base config that I get new devs in the NY office is:

  • Dell Optiplex 980 class tower
  • Max out the RAM
  • Best i7 Processor that I can get in it
  • SSD primary drive
  • 7.2k large (500GB – 1TB) Secondary Drive
  • 30″ Primary Monitor
  • 20″ Secondary monitor (turned sideways)

Every dev picks their own keyboard – we may give them the crappy one that comes with the machine to get them up and running, but they can request any keyboard/mouse combo they want (and … i do mean any).

The not so scalable

  • For our remote devs we get workstation replacement class laptops. Basically I go out and find the most powerful laptop I can get at the time they start, as well as a 30″ monitor to go with it. Not Scalable.
  • At least one dev in the NY office has a fully customized hand built machine with 2 30″ monitors. VERY not scalable.
  • Whatever they want – within reason

Basically, it boils down to get them what they need to get their job done. One of the biggest challenge to some people is that they do not have management’s buy-in to get the Devs what they need. I find that this is a sad state of affairs, but I have no real advice to offer – especially since I work somewhere that management has mandated great dev machines.

We recently changed the NICs in our web tier and primary database servers from Broadcom to Intel based NICs based on some … issues we had been having. After we put them in they worked reasonably well, but we knew that they could be faster and push more data. When I started to dig into just what we could do to tweak the pleathora of settings for the new NICs I found a few settings that would probably help a little bit, and one technology that could help us out tremendously.

Changes at a glance

  • Turned on Intel I/OAT
  • Adjusted Send and Recieve buffers to 2048 (max allowed)
  • Turned off interrupt moderation
  • Increased the Receive Side Scaling Queues from 1 to 4

The long version

The first and biggest change is that we turned on Intel’s I/OAT technology. It’s a collection of different techniques that work together to improve the performance of your host networking, as defined on Intel’s website:

  • Intel® QuickData Technology — enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput.
  • Direct Cache Access (DCA) — allows a capable I/O device, such as a network controller, to place data directly into CPU cache, reducing cache misses and improving application response times.
  • Extended Message Signaled Interrupts (MSI-X) – distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance.
  • Receive Side Coalescing (RSC) — aggregates packets from the same TCP/IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing.
  • Low Latency Interrupts — tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency.

The catch is you need to be running a full Intel hardware stack. Your CPU, Motherboard, BIOS, NIC and OS all need to be compatible with the technology to be able to use it. Once you have the right stack in place, you might have to turn on a BIOS option, but that’s it no tweaking or poking to make it just right, it’s just right out of the box. Turning it on was as simple as flipping a BIOS setting, aptly named “Intel I/OAT.”

Performance tuning options

The performance options are all exposed via the PROSet utility making it nice and easy to change, no need to go digging into the registry for some esoteric key that may or may not be there. For each of them there is a trade off you need to consider. Some of the options will increase host CPU, some will cause higher host memory usage. To find the right value for your systems you really need to evaluate your overall situation and see if the trade offs are worth it.

Send and Receive Buffers

The Send and Receive buffers where set to the maximum allowed value of 2048. The trade off here is that you will consume more host memory. For us this is not a big deal since we have a lot of RAM on our boxes. Also, we had been seeing a good deal of Zero-window TCP packets when investigating our network so we needed to increase the buffer anyway.

Interrupt Moderation

The Interrupt Moderation feature was disabled. This feature allows you to have the NIC throttle the number of interrupts to the CPU which will limit the number of CPU cycles used by the NIC interrupts. Turning this off will increase your CPU usage, but it will also prevent packets from sitting there waiting for an interrupt to be proccessed. The increased cpu is a pain point for us right now (we are working on fixing that) but I believe it’s worth it.

Receive Side Scaling Queues

Receive Side Scaling is a technology that allows you to process a TCP connection across multiple cores. This allows for more efficient cache and processor usage when you TCP connection is not tied to a single core. When you are using this feature it will only use real physical cores to process TCP connections, you are not able to use this with hyper thread cores.

Receive Side Scaling Queues are essentially buffer space that is used between the NIC and the CPU when you are using Receive Side Scaling. This is another setting that has a trade off between host CPU and performance. I opted for the trade off, and increased the queues from 1 to 4 queues.

Additional notes

  • Since I/OAT needs BIOS support and the BIOS on our web tier was woefully out of date anyway I updated to the latest BIOS for those machines
  • I updated the Intel PROSet utility to verion 16.1 from 16.0

We are less than twenty four hours into using these new setting, but everything looks much much better through our peak traffic today. So far we are very happy with the results of these changes.

We’ve been doing a lot of talking about some of the growing pains we have been having with our database server. And how we are going to deal with it. Well we will be moving Stack Overflow’s database to its own hardware this Friday (2011-03-11).

When?

2011-03-11 9:00pm Eastern (2011-03-12 02:00 UTC)

Why?

To Make everything MORE AWESOME

What?

2 new Dell R710′s

  • 2x Intel Xeon Processor X5680 3.33 GHz
  • 96GB of RAM
  • 6 Intel X25-E SSD’s in a RAID 10

What will this mean?

First, we will be putting Stack Overflow into Read Only mode again for the maintenance – just like we did when we went down the Oregon trail. Once the migration is complete, Stack Overflow will be on its own DB pair, and can run free, and the rest of the Stack Exchange sites don’t have to contend with the Megalopolis for database time, making them MORE AWESOME as well.

I will be upgrading the Stack Exchange network’s switching infrastructure on Saturday February, 26th. There should be minimal downtime while I do this, however the sites may be a little slow as I work through the Web Tier. Additionally there will be about a 10-20 minute complete site downtime while I move the DB servers and routers.

What exactly will I be doing? Why, I’ll be removing our current Dell PowerConnect 5448 switches and replacing them with Cisco 2960-S-48TS-L switches.

The plan is to start working on moving the web tier Saturday afternoon and be finished moving all services by early evening. I will post to Blog.Serverfault about one hour before the start of work.

I thought as a break form the normal prose some of our readers might enjoy a short overview of the Stack Exchange Network (including Stack Overflow, Server Fault, and Super User) from a technical view:

Traffic:

  • 95 Million Page Views a Month
  • 800 HTTP requests a second
  • 180 DNS requests a second
  • 55 Megabits per second

Data Centers:

Production Servers*:

  • 12 Web Servers (Windows Server 2008 R2)
  • 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
  • 2 Load Balancers (Ubuntu Server and HAProxy)
  • 2 Caching Servers (Redis on CentOS)
  • 1 Router / Firewall (Ubuntu Server)
  • 3 DNS Servers (Bind on CentOS)

Software and Technologies Used:

Developers and System Administrators:

  • 14 Developers
  • 2 System Administrators

*(excludes fail over and management servers)

Our Storage Decision

George Beech

We’ve been trying to figure out what to do about the Disk subsystem IO problems we have. We spent many hours talking to vendors, trusted advisers (ok, reaaaally smart people who were willing to help us – and put up with our questions), and each other trying to find the best solution. We think we have, for now.

Before I get into what we decided to do I want to talk a little about our thought process and the requirements we put together to shape our decision.

Requirements

It should be fast

I know this may seem silly but we pride ourselves on the fact that our sites load not just fast but really fast. We needed a solution that didn’t just get us to the point where our storage subsystem was good, but decimated that barrier and made it great. The added benefit to this approach is we should be able to handle the IO needs of our sites for at least the next year or so. (Assuming our growth rate projections are accurate, of course.)

It should be reasonably priced

The primary requirement of this upgrade was to get more speed out of our storage subsystem, so it makes no sense to spend $100k or more on a SAN with a whole bunch of features that we won’t use in the next 6-12 months. But we don’t want to spend the least amount of money we possibly can on equipment, either. Like Goldilocks, we needed to find a solution that was just right.

It should be reasonably safe

By safety, I mean our data won’t go up in smoke when there is a catastrophic failure — and it needs to keep our services running briskly without a ton of overhead, either mental or physical.  We’re not a bank, and we don’t need to over-build our systems to never lose a single bit and always be 100% accurate. But we also respect that you put a lot of effort into making this place great, and we owe it to you to treat the content you’ve contributed with care.

Putting aside all the cool stuff, all the other feature options, does spending X on storage technology Y give us the best price/performance?

Research

The first thing we did is to explore the different classes of server storage, and look at the pros and cons of each.

1. Direct Attached Storage enclosures

Pros: Fairly low cost, although more expensive than internal storage. You get a large number of spindles to run faster with.
Cons: Normally can only be connected to a low number of hosts (1-4), for what you get, it’s a large investment.

2. SAN technologies

Pros: Flexible storage, generally expandable (for a price), lots of disks.
Cons: Very expensive. And to get the best performance we would need an infrastructure upgrade as well, either 10 Gigabit Ethernet or Fibre Channel.

3. PCI bus based flash drives (FusionIO)

Pros: ridiculously fast. not a horrible price. Did i mention F-A-S-T FAST?
Cons: relatively new tech, giant single point of failure with no great way to compensate for it.

4. Traditional Solid State Hard Drive (SSD) storage

Pros: Fast, potentially really fast. flexibility for future upgrades without sinking too much money into one technology.
Cons: Have to get non-vendor “approved” drives, since our vendor wanted insane amounts of money for a single drive.

Evaluation

We had grandiose plans to bring a few of the finalist options in house to do a barrage of tests, but this turned out to be harder than we thought. Storage vendors were really reluctant to allow us to bring in demo units of their hardware. When I’m looking to spend tens of thousands of dollars, I’m sorry, but I really want to either a) go to your lab and actually run test against a unit that I’m going to be buying, or b) bring a unit in house for a few weeks or a month and be able to verify that the unit will do what you claim it will. What actually happened was that we only brought in one option: the Intel X25-E solid state hard drive. The price wasn’t too expensive to bring them in on a flier, and if they didn’t work out we would be able to re-purpose them somewhere else so it wouldn’t be a total loss.

We decided to use Brent Ozar’s benchmark that he did on the FusionIO drives as our benchmark. Brent isn’t some random blogger, you understand — he is a database ninja. When he does a benchmark, he does a benchmark.

Here’s a quick comparison of our results and his:

Random Reads — 2 threads, 8 outstanding requests, 64k blocks

FusionIO Intel X25-E
in RAID10
MB/s 1424 1064
IOs/s 22788 17023

Random Writes — 2 threads, 1 outstanding request, 64k blocks

FusionIO Intel X25-E
in RAID10
MB/s 632 584
IOs/s 10114 9337

The numbers for the X-25′s are actually limited by the controller since the H700 has a max throughput of 600MBps. We do not know how much faster this setup could go if we put a higher throughput controller in the boxes.

We ended up not bringing in any of the SANs due to the amazing performance we got out of the Intel X25 solid state drives. Speed was all we were really after. We don’t seen the need for a SAN at this point so we’ll let that money continue to earn interest in the bank for now.

The full test that we ended up running was straight out of – you guessed it – Brent Ozar’s playbook. That us, using SQLIO to run a whole barrage of tests against the storage system.

The test system consisted of:

  • Dell R710 with 96GB of memory
  • 2 Xeon X5680 CPUs
  • 6 Intel X25-E drives in a RAID 10 array
  • H700 RAID controller with 1GB of memory

The gold standard for fast is the FusionIO drives. In our benchmarking, a RAID 10 array of Intel X25 drives got within about 25% of FusionIO performance for reads, and within 10% for writes. That exceeded everyone’s expectations and made our decision very easy. Six X25-E’s are about half the list price of one FusionIO, and putting them in a RAID array eases any concerns about reliability.

For all who are interested the raw data can be found in this archive. The archive also includes a pretty ugly Perl script that converts the raw output from SQLIO into a csv file.

> I would like to take a moment to thank all the people who helped us out looking at all these options. > > – Our vendors who put up with insane questions and flip floppery – Dell, CDW, and Fusion-IO. > – All of the regulars in the Server Fault chat room. > – Of course Brent Ozar who … is … godlike? Hmm, let’s just go with “great guy”.