Hey everyone, I’m one of the newer ServerFault sysadmins and I’ve got the fun job of explaining why we have been moving to Windows Server 2012 just almost as fast as WDS can deploy images.


What’s Wrong With Windows Server 2008 R2?

Where do I begin?

2008 R2 was a great operating system in its day.  Automation was becoming a core part of the platform.  PowerShell was shipped in the box, enabled by default, and supported several roles and features.  Many of the kinks were worked out after the Vista/2008 release cycle and people were starting to acclimate to the User Account Control (or figured out how to turn it off).

Server 2008 R2 held up well for a long time, but was starting to show it’s age with our SQL Server cluster.  There are a bunch of hotfixes required to make clustering have a chance of working with SQL Server 2012 AlwaysOn in our environment.  Brent Ozar, in part based on experiences helping us, now recommends not building AlwaysOn Availability Groups on Server 2008 R2.

Additionally, the management story for Windows Server up to this point has been very focused on single machines.  Server Manager in 2008 R2 targets one machine at a time.  Most of the MMCs deal with one machine at a time.  PowerShell remoting was not on be default.  It is definitely work to build a multiple machine management story.


So, how does Server 2012 help?

First of all, Windows Server 2012 helps address our major pain point of how SQL Server 2012 Availability Groups interact with Windows Clustering.  Clustering received a lot of attention in Windows Server 2012.  The addition of dynamic quorum helps maintain the cluster state in the case of site-to-site VPN interruptions.  Clustering itself has just become more resilient and offers a wider feature set.

The second (and I think most important feature) that Windows Server 2012 offers is SMB3.  SMB3 fixes many of the performance considerations with access to files stored centrally on a file share.  SMB Multichannel allows SMB clients and servers to leverage multiple network connections, makes SMB transfers resilient to TCP disconnects (by having multiple streams per transfer, even with one NIC), and provides the foundation for scale out file servers, providing a way for multiple servers to share the same connection concurrently, isolating storage transactions for single server failures and allowing connections to scale across multiple servers.  SMB3 provides a number of opportunities to radically change the structure of your application data access and I talked with Richard Campbell about this at length on RunAs Radio.

The third huge win is how Windows Server 2012 supports three levels of GUI-ness.  You have the traditional full UI to support the traditional administrative experience, and since this is the same codebase as Windows 8, you get the Metro/Modern UI start screen as a “bonus”.  The next level of GUI-ness offered by Server 2012 is the minimal shell experience, or what I like to call “Server Core with Training Wheels”.  The minimal shell removes Explorer, Internet Explorer and most GUI apps, but leaves behind Server Manager and the MMCs.  This provides a bit of a security blanket for those admins who aren’t quite ready to give up managing a server locally, but want to break their reliance on the UI and grow up to be SYSTEMS administrators who can manage multiple systems.  By removing explorer.exe and iexplore.exe, you’ve eliminated a huge source of patches, particularly the ones that require a reboot.  Additionally, since there is no Explorer, the RDP experience is a command prompt and Server Manager – discouraging random RDP sessions.  The final level of GUI-ness is full on Server Core.  This is the DEFAULT install option for Windows Server 2012.  Server Core lacks even the minimal GUI tooling provided by the minimal shell configuration.  Server Core, as of Server 2012, is no longer a life choice.  In Server 2008 and Server 2008 R2, when you selected Server Core as the install option, you were stuck in Server Core.  With Server 2012, the GUI bits are features and can be added and removed through the standard feature management tooling.  I talked a bit about this with Richard Campbell on RunAs Radio earlier this year.

Next up in the “pro” column for Windows Server 2012 is PowerShell V3 and the increased cmdlet coverage in the operating system.  PowerShell v3 brings a number of great enhancements to the party like improved performance, resilient remoting sessions, commands for working with the web and REST endpoints, workflow, ISE enhancements and more.  Windows Server 2012 also brought about 2400 commands across the in-box roles and features (up from under 500 in 2008 R2).  This explosion in coverage was due to improvements in the WMI APIs and a new API in PowerShell for being able to take an XML mapping file and generate PowerShell commands from a WMI API.  This got us tons of additional coverage, including the network stack. Since servers really aren’t intended to be managed individually (despite what some sysadmins, product teams, and third parties software vendors believe), the increased PowerShell coverage makes that a much more reasonable task.  If you are interested in further talk about PowerShell V3, you guessed it, there was a RunAs Radio talking about that as well.

The final consideration we had for choosing Server 2012 was in-box NIC teaming.  Server 2012 offers OS level NIC teaming with a variety of teaming options.  Given the problems we’ve faced with third-party NIC teaming (http://blog.serverfault.com/2011/03/04/broadcom-die-mutha/ and some other fun stories), I was gung-ho to get switched over to the native teaming features (which I’ve monitored in production environments for over a year at that point).  Having the network load-balancing and failover (LBFO) teaming included in the OS is a major plus.  Don’t worry, I won’t point you to yet another RunAs Radio show this time.


Where are we now?

I’m currently working with our team to continue the roll-out of Server 2012.  Part of that process is updating our installation and configuration management plan.  Nick Craver, one of our developers who likes to help with the sysadmin work and has been featured on this blog, and I have spent a good amount of time tweaking our deployment scripts (server and application deployment) to facilitate the roll out of new servers.  We’ve got our SQL Server and web tier installs fully scripted, to the point where we can go from bare metal to a fully functional web server in about 3 hours (mainly OS install time and running Windows Updates).

We are still working out the rest of our deployment story, looking for integration points with our Puppet configuration management, making the configuration idempotent, and creating artifacts that can be stored in source control to version our environment.  But each week sees more Server 2012 (and PowerShell use…  Pete’s started using a bunch of stuff I wrote and now I have bugs to fix… grrr.. ) in our environment.


Where do we want to be?

I would like to see our Windows deployments that are in a production capacity (internally or externally facing) under full configuration management control, with changes pushed via a configuration management infrastructure and configurations enforced regularly.

I would like to have most of our servers running Server Core (doesn’t make much sense to do that for our management stations, but for most servers it works).

Most of all, I would like our server deployment to help serve as a template for a different way of Windows administration, one with rigid standards but quick to respond to changing needs.  I’d like to see our infrastructure as agile as our code base, where changing our OS configuration is as easy as pushing a build is for our development teams.  I’d like to be able to validate infrastructure changes quickly, closing the feedback loop and allowing worry free infrastructure changes.

You all have a front row seat to our environmental changes and I plan to blog extensively about my configuration management efforts in the near future, giving you an opportunity to learn from my mistakes.


An update on Sandy

Nick Craver

We’re still up and running in Oregon, and have powered down all services in our New York datacenter at 75 broad.

While we’re not actually running anything out of the datacenter, here’s what’s happening:

After our last update the Peer1 datacenter continues to stay online any way they can.  When Sandy hit, their basement fuel store was contaminated and the pumps necessary to get it up 17 stories to the generator went offline.  Since then employees from several companies hosted there have shown up and continue fueling the generator via bucket brigade.

Our sister company Fog Creek has many hands down there right now, you can follow their status updates here.  Squarespace is in the exact same boat as well, you can see their updates here.

The concern has been fuel supply to carry upstairs, as I’m writing this a truck has pulled up and the barrels are filling.  Several of those companies with people on the ground have been fetching fuel from gas stations for the past few hours.

While this is happening the basement is being pumped out so that the facility can be repaired and grid power restored.  We don’t have an ETA at this time.

We’ll try and post an update in a few hours.

Our New York Peer 1 datacenter at 75 Broad is still running on generator power, but as a precaution we decided to failover Stack Overflow, Careers, and the rest of the Stack Exchange network to our secondary datacenter in Oregon last evening. It turned out to be the right call because the refueling trucks can’t get to the facility, so Peer 1 is shutting down all power in about 30 minutes.

We actually recently tested a lot of this, but this is our first time failing over everything at once. So far it is going pretty well, but we have run into a few issues so far:

  • An index reorg job kicked off right before failing over. This meant that our SQL replication partners across the country were 40 Gigabytes behind. So Stack Overflow had to remain in read only for about an hour
  • Because the status message on our sites is stored in the database, that was readonly, so we couldn’t update it to let everyone know it would be read only for about an hour
  • We realized we have to transfer the AD FSMO role forcefully since the NY DCs were shut down, and we don’t know how much fuel is left
  • Our backup monitoring system isn’t permitted as an SNMP manager via the Group Policy, so we have to update that

We have some open concerns, and will be keeping a close eye on the following:

  • Oregon has some lower end Dell switches, we hope they handle the load. We will be shipping the current 2960S switches to OR once we upgrade our NY switches to the Nexus 5k/2k line in a couple of weeks
  • Our load balancers out in OR are a little tight on CPU
  • We have 5 web servers in OR instead of 10. However, the combined CPU load of NY on the web tier is ususally 100-200% (Out of 1000%) so I think it will be okay:

However, in the big picture, we have successfully failed over to Oregon! Today is going to feel like taking the sub below its depth rating, you can watch the Das Boot Video to share our feelings.

This coming Saturday, October 13th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure.  We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of.  We’re planning for mass chaos and lots of fail.

Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon.  Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.

So here’s what we want to happen this weekend:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode
  4. Find problems, squash bugs in our configuration until we’re running smooth
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode
  4. Get beer

What will happen? We don’t know – but we’ll blog here about it.  The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week).  The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.

All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.

Here are a few things we’re afraid (again, why we’re testing):

  • Internal API wonkiness while we’re switching DNS (our sites talk to each other)
  • Bad DNS caching on misconfigured servers/proxies
  • SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
  • 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
  • Oregon doesn’t have enough upstream bandwidth

Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.

At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.

Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?

Why Use Virtualization?

Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.

This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:

  • Live migration (vMotion in VMWare terms)
  • Running multiple operating systems (i.e. Windows and Linux) on the same hardware
  • Easier to get full utilization of hardware resources by moving VMs around

These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?

The Joy of Commodity Hardware

As Wikipedia defines it:

Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”

Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.

So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.

Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.

Virtualization and Centralized Storage are Best Friends

With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.

Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.

Virtualized Clusters to the Rescue?

If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.

VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.

Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.

What Will We Go With?

Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.

I’ve recently been looking back on what we have written about our architecture in the past, and came to a stunning realization. That realization is that while we have many many different articles about what we have been doing there hasn’t been a good, solid overview of our architecture in a long time. In fact, the last really comprehensive write-up was done by Jeff before this blog even existed. And, boy I do have to say there has been quite a lot of change behind the scenes since then. So, my dear readers I’m going to take some time – and my next few blog posts – to give everyone an in depth look into how we have the Stack Exchange Network setup to serve between 12 and 14 Million page views per day.

How these posts will breakdown

Since we have obviously grown, and are offering more services to our users I’m going to break these posts out by each of the 4 major services we offer to our user base:

  • Core Q&A (this includes the API)
  • Careers
  • Chat
  • Community Blogs

Each one of these systems all work towards our goal of making the internet better, but they have different requirements and different challenges.

In this first post, I’ll be focusing on our core Q&A system, since that is after all our bread and butter.

Core Q&A

First, a high level overview of how everything is put together:

The Hardware

Our core hardware setup hasn’t changed all that much. Well, I should say the chassis haven’t changed that much. We’ve done a lot of work to upgrade the internals of the servers when needed to address performance issues as they came up, as well as handle issues that resulted from Stack Overflow being so big.

Web Tier

Of these 10 Servers, 3 are dedicated to Stack Overflow with an additional 3 servers serving Stack Overflow and the Stack Exchange Network. We have one server dedicated to Dev/QA – which also hosts meta.stackoverflow.com. Our Web Tier machines normally operate between 5 and 20% utilization. We have plenty of room to grow on these boxes.

  • 10 Dell R610 IIS web servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
    • 16 GB RAM
    • Windows Server 2008 R2
    • 2 drives
      • RAID 1
      • 2x Intel 320 300GB SSD (RAID 1)

DB Tier

We have two database server pairs. One pair is dedicated to running Stack Overflow, and the other runs the rest of the network. We run development against the secondary server of the non-stack overflow database pair. Both of our database pairs run at about 20% utilization, so once again we have room to grow here as well.

  • 2 Dell R710 database servers:
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 spindles
      • Mirrored Pair for OS
      • 6 disk RAID10 for databases
    • SQL Server 2008 R2 SP1
  • 2 Dell R710 database servers (Stack Overflow Dedicated):
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 drives
      • Mirrored Pair for OS
      • 6 drive RAID10 of Intel X25-E SSDs for Database
    • SQL Server 2008 R2 SP1

Caching Tier

We run redundant Redis servers for our caching tier.

  • 2 Dell R610 Redis servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz
    • 16 GB RAM
    • CentOS

Network Layer

We use HAProxy for our load balancing, and Cisco Switching.

  • 2 Dell R610 HAProxy servers:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 4 GB RAM
    • Ubuntu Server
  • 6 WS-C2960S-48TS-L Gigabit switches
    • FlexStack (two stacks, 4 switches and 2 switches)

Data Integrity

As with any system, making sure that your data is backed up and the backups are good is an integral part to your service offering. We backup our databases nightly and restore them to two different locations. One local to our NY data center for our devs to work against, and one remote in our OR data center.


Overall I believe that we are in a good place and have plenty of room to grow given our current setup. As always we will constantly be looking at our infrastructure and tweaking it to get the best performance possible and give our users the best experience possible.

A couple of weeks ago we had one of our edge routers go down on us. Nothing bad happened, failover to our secondary router work just as expected. Now, we saw something wierd this week when we looked at the internal interface graph for our secondary router.

I’ll give you a second to try and see what we saw – although I don’t think you’ll need a whole second to see something very very strange going on with this router. That’s right, there is a whole lot of outbound traffic on this router, but ZERO inbound traffic. The next question we had was what could possibly be causing this? I really don’t think that I could be anything good.

You can clearly see on the graph where we failed back to our primary router. But, after that there is still a ton of traffic that is traversing our secondary router when there should be very little traffic going through there.

After some digging around we found that our Windows servers had the wrong ARP address for the VIP of our routers. That’s right, Windows still had the wrong ARP address after the fail back to the primary router, it even had the wrong ARP address days later.

How could this be possible? I was stumped after some digging around it seems that Microsoft changed the way that the network stack handles Gratuitous ARP packets (GARP packets) with Windows Vista/2008 RTM. This change has persisted through to Windows 7 and Windows 2008 R2.

What is Gratuitous ARP?

Gratuitous ARP is when a system sends out an ARP packets announcing to all system what it’s MAC address is. You generally see these in HA environments that make use of Virtual IPs that can move back and forth between machines. You will normally see a machine issue a GARP packet when a fail over event occurs and the new machine picks up the VIP. Wikipedia ARP article for more info on the ARP protocol

What did Microsoft Change?

There is actually very little information out there about windows and how it handles GARP. The best resource i’ve found that gives a very good overview of the new windows networking stack is a very well written technet blog. About 3/4 of the way down there is a section named “Changes to ARP cache updating” within this section lies the answer to all the mystery of our weird network bandwidth.

>First, a Windows Vista or Windows Server 2008 will not update the Neighbor cache if an ARP broadcast is received unless it is part of a broadcast ARP request for the receiver. What this means is that when a gratuitous ARP is sent on a network with Windows Vista and Widows Server 2008, these systems will not update their cache with incorrect information if there is an IP address conflict.

>Additionally, when a gratuitous ARP is sent by a Windows Vista or Windows Server 2008, the following change has been made – the SPA field in the initial request is set to This way the ARP or neighbor caches of systems receiving this request are not updated. So, if there is a duplicate IP address, the receivers do not need to have their cache corrected.

The question is why is this such a big problem for Microsoft? Well the answer is they have hi-jacked the GARP packets for their Address Conflict Detection mechanism. You know that pop-up that says “Another machine on this network has been detected with the same IP address”. With previous version of windows they had the same mechanism, but still respected the normal GARP packets, thus there would sometimes be an issue with Windows systems updating their ARP cache with invalid data. They fixed it by breaking GARP.

Why is this a problem?

Beyond the issue we have seen with Windows 2008 not respecting GARP packets this can cause other wierd problems. One example I can think of off the top of my head, is that for HA systems that use GARP to facilitate moving the VIP when a system goes down is that you will now have to wait for the OS to timeout the neighbor cache. This will add more time to your fail over, possibly causing things that are expecting a quicker fail over to break.

Is there a Fix?

I have not been able to find a fix for this. Although there is very little information out there on Windows networking at that low of a level. If you know of a fix to this issue I’ve started a question on Server Fault.