Hey everyone, I’m one of the newer ServerFault sysadmins and I’ve got the fun job of explaining why we have been moving to Windows Server 2012 just almost as fast as WDS can deploy images.


What’s Wrong With Windows Server 2008 R2?

Where do I begin?

2008 R2 was a great operating system in its day.  Automation was becoming a core part of the platform.  PowerShell was shipped in the box, enabled by default, and supported several roles and features.  Many of the kinks were worked out after the Vista/2008 release cycle and people were starting to acclimate to the User Account Control (or figured out how to turn it off).

Server 2008 R2 held up well for a long time, but was starting to show it’s age with our SQL Server cluster.  There are a bunch of hotfixes required to make clustering have a chance of working with SQL Server 2012 AlwaysOn in our environment.  Brent Ozar, in part based on experiences helping us, now recommends not building AlwaysOn Availability Groups on Server 2008 R2.

Additionally, the management story for Windows Server up to this point has been very focused on single machines.  Server Manager in 2008 R2 targets one machine at a time.  Most of the MMCs deal with one machine at a time.  PowerShell remoting was not on be default.  It is definitely work to build a multiple machine management story.


So, how does Server 2012 help?

First of all, Windows Server 2012 helps address our major pain point of how SQL Server 2012 Availability Groups interact with Windows Clustering.  Clustering received a lot of attention in Windows Server 2012.  The addition of dynamic quorum helps maintain the cluster state in the case of site-to-site VPN interruptions.  Clustering itself has just become more resilient and offers a wider feature set.

The second (and I think most important feature) that Windows Server 2012 offers is SMB3.  SMB3 fixes many of the performance considerations with access to files stored centrally on a file share.  SMB Multichannel allows SMB clients and servers to leverage multiple network connections, makes SMB transfers resilient to TCP disconnects (by having multiple streams per transfer, even with one NIC), and provides the foundation for scale out file servers, providing a way for multiple servers to share the same connection concurrently, isolating storage transactions for single server failures and allowing connections to scale across multiple servers.  SMB3 provides a number of opportunities to radically change the structure of your application data access and I talked with Richard Campbell about this at length on RunAs Radio.

The third huge win is how Windows Server 2012 supports three levels of GUI-ness.  You have the traditional full UI to support the traditional administrative experience, and since this is the same codebase as Windows 8, you get the Metro/Modern UI start screen as a “bonus”.  The next level of GUI-ness offered by Server 2012 is the minimal shell experience, or what I like to call “Server Core with Training Wheels”.  The minimal shell removes Explorer, Internet Explorer and most GUI apps, but leaves behind Server Manager and the MMCs.  This provides a bit of a security blanket for those admins who aren’t quite ready to give up managing a server locally, but want to break their reliance on the UI and grow up to be SYSTEMS administrators who can manage multiple systems.  By removing explorer.exe and iexplore.exe, you’ve eliminated a huge source of patches, particularly the ones that require a reboot.  Additionally, since there is no Explorer, the RDP experience is a command prompt and Server Manager – discouraging random RDP sessions.  The final level of GUI-ness is full on Server Core.  This is the DEFAULT install option for Windows Server 2012.  Server Core lacks even the minimal GUI tooling provided by the minimal shell configuration.  Server Core, as of Server 2012, is no longer a life choice.  In Server 2008 and Server 2008 R2, when you selected Server Core as the install option, you were stuck in Server Core.  With Server 2012, the GUI bits are features and can be added and removed through the standard feature management tooling.  I talked a bit about this with Richard Campbell on RunAs Radio earlier this year.

Next up in the “pro” column for Windows Server 2012 is PowerShell V3 and the increased cmdlet coverage in the operating system.  PowerShell v3 brings a number of great enhancements to the party like improved performance, resilient remoting sessions, commands for working with the web and REST endpoints, workflow, ISE enhancements and more.  Windows Server 2012 also brought about 2400 commands across the in-box roles and features (up from under 500 in 2008 R2).  This explosion in coverage was due to improvements in the WMI APIs and a new API in PowerShell for being able to take an XML mapping file and generate PowerShell commands from a WMI API.  This got us tons of additional coverage, including the network stack. Since servers really aren’t intended to be managed individually (despite what some sysadmins, product teams, and third parties software vendors believe), the increased PowerShell coverage makes that a much more reasonable task.  If you are interested in further talk about PowerShell V3, you guessed it, there was a RunAs Radio talking about that as well.

The final consideration we had for choosing Server 2012 was in-box NIC teaming.  Server 2012 offers OS level NIC teaming with a variety of teaming options.  Given the problems we’ve faced with third-party NIC teaming (http://blog.serverfault.com/2011/03/04/broadcom-die-mutha/ and some other fun stories), I was gung-ho to get switched over to the native teaming features (which I’ve monitored in production environments for over a year at that point).  Having the network load-balancing and failover (LBFO) teaming included in the OS is a major plus.  Don’t worry, I won’t point you to yet another RunAs Radio show this time.


Where are we now?

I’m currently working with our team to continue the roll-out of Server 2012.  Part of that process is updating our installation and configuration management plan.  Nick Craver, one of our developers who likes to help with the sysadmin work and has been featured on this blog, and I have spent a good amount of time tweaking our deployment scripts (server and application deployment) to facilitate the roll out of new servers.  We’ve got our SQL Server and web tier installs fully scripted, to the point where we can go from bare metal to a fully functional web server in about 3 hours (mainly OS install time and running Windows Updates).

We are still working out the rest of our deployment story, looking for integration points with our Puppet configuration management, making the configuration idempotent, and creating artifacts that can be stored in source control to version our environment.  But each week sees more Server 2012 (and PowerShell use…  Pete’s started using a bunch of stuff I wrote and now I have bugs to fix… grrr.. ) in our environment.


Where do we want to be?

I would like to see our Windows deployments that are in a production capacity (internally or externally facing) under full configuration management control, with changes pushed via a configuration management infrastructure and configurations enforced regularly.

I would like to have most of our servers running Server Core (doesn’t make much sense to do that for our management stations, but for most servers it works).

Most of all, I would like our server deployment to help serve as a template for a different way of Windows administration, one with rigid standards but quick to respond to changing needs.  I’d like to see our infrastructure as agile as our code base, where changing our OS configuration is as easy as pushing a build is for our development teams.  I’d like to be able to validate infrastructure changes quickly, closing the feedback loop and allowing worry free infrastructure changes.

You all have a front row seat to our environmental changes and I plan to blog extensively about my configuration management efforts in the near future, giving you an opportunity to learn from my mistakes.


Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.


We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

  • Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
  • Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
  • Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
  • Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

  • There are constraints of what is actually available to buy, and how much you can actually know what you are getting
  • Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.


When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

  • Gave us time to rectify some mistakes
  • Gave us confidence in our secondary datacenter since we failed over to it before the storm
  • Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.


Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

The move to SQL 2012

Nick Craver

Sorry this blog has been a bit quiet lately, we’ve been very busy making some big changes behind the scenes.  So what are we up to?  Let’s start with just the SQL infrastructure moves, here’s a list of servers in play as they started out:

  • NY-DB01 – SQL2008 R2 Hosts all sites except Stack Overflow and the Sites DB
  • NY-DB02 – SQL2008 R2 Daily backups restored from NY-DB01, and the Dev DBs
  • NY-DB03 – SQL2008 R2 StackOverflow’s DB
  • NY-DB04 – SQL 2008 R2 StackOverflow 5 minute behind hot spare in restore mode
  • OR-DB01 – SQL2008 R2 Chat’s DBs
  • OR-DB02 – SQL2008 R2 SEDE, Internal SEDE and Chat Dev DBs
  • New Dell R710 w/ 2x OS Drives + 6x Data 300GB Intel 320 SSDs in RAID10 and 96GB RAM
  • 3xNew Dell R720 w/ 2x OS Drives +  12x Data 200GB Intel 710 SSDs in RAID10 and 384GB RAM

First we set up the first SQL2012 Cluster with those new R720 machines. The new R720s are identical; they became NY-SQL01, NY-SQL02 and OR-SQL01:

  • NY-SQL01
    • Primary: Sites DB, Stack Overflow
    • Backups: Sites DB & StackOverflow Full and Transaction Logs -> NY
  • NY-SQL02
    • Replica: Sites DB, Stack Overflow, Chat DBs
    • Backups: Chat DBs Full -> NY
  • OR-SQL01
    • Primary: Chat DBs
    • Replica: Sites DB, Stack Overflow
    • Backups: Sites SB, StackOverflow & Chat DBs Full -> OR

For this we have 2 new availability groups, StackOverflow_AG and Chat_AG. The primary server for StackOverflow_AG is NY-SQL01 replicated to a secondary in the same data center (NY-SQL02) and across the country to Oregon (OR-SQL01).  The Chat_AG is only on 2 servers: the OR-SQL01 primary (chat is hosted in Oregon) and the replica NY-SQL02. The reason chat is only on 2 servers is because SQL2012 availability groups do not have the ability to distinguish between sites and replicate that way…so it would send the same transaction stream across the country twice to replicate to the NY servers, rather than echoing the transactions through one to the other…this is an unnecessary use of bandwidth we feel.

The StackOverflow and Sites DB portion of the first cluster was completed on the 2012-08-11 maintenance window; chat will be completed on 2012-08-18 (part of chat has moved, we want to give it a week to observe any problems).  Now what happens in the following week? We need to shuffle some hardware around.

With the StackOverflow DB moved off of the NY-DB03 and NY-DB04 pair, they’re ready to be re-tasked.  Currently these servers are identical Dell R710s with 288GB of RAM, 2x OS Drives in a RAID 1 and 6x 200GB Intel 710 SSDs in RAID10.  These boxes get re-tasked to be NY-SQL03 and NY-SQL04. Joining them in this second SQl2012 cluster is OR-SQL02, that new Dell R710 above.  Here’s a breakdown:

  • NY-SQL03
    • Primary: All Stack Exchange 2.0 Sites other than SO
    • Backups: SE 2.0s Full + [diff of trans] -> NY
  • NY-SQL04
    • Primary: SE 2.0 & SO Dev DBs
    • Replica: All Stack Exchange 2.0 Sites other than SO
  • OR-SQL02
    • Primary: Chat Dev DBs
    • Replica: All Stack Exchange 2.0 Sites other than SO
    • SE 2.0s Full + [diff of trans] -> OR

Now we’ve freed up the NY-DB01 and NY-DB02 boxes, they’ll also be nuked, get some new drives and be re-tasked for some other purposes (for example, one goes to Oregon to be the HAProxy traffic log out there).

For the miscellaneous bits, OR-DB01 will be freed up after the move to OR-SQL01 and OR-SQL02 of the chat DBs.   We’ll then take OR-DB01 and install 2012 re-tasking it to host the data.stackexchange.com databases.  It has double the memory of the current server and should provide a nice boost to performance there.


Why? What does all this moving get us?  Well it turns out SQL 2012 Always on Availability Groups give us quite a bit.  Here are the big ones for our architecture:

  • Near real-time replicas of every production database, ready to go
  • No more copying backups across to the offsite datacenter for redundancy
  • We can read from the replicas, eliminating the need for an entire server and allows us to spread the read load out (e.g. API can point at a replica)
  • A backup DR location is now doable

First, we can have very near real-time hot spares for all production databases (previously, we had up to 8 hours data loss between differentials).  Second, we don’t need to do these wasteful copies of databases across the country purely for backup purposes…we have a nearly-in-sync replica across the country we can do speedy local backups from.  That’s a huge cross-country VPN bandwidth savings as an added bonus.  Third, we can spread the read load out across multiple servers (and we can add another 2 more to either of these availability groups if needed).  Performance-wise, we don’t even have a need for any read load spreading, but it’s very nice to have as an option. Now for the last one: a DR site.

PEAK Internet in Oregon is where Stack Overflow began on a single server, and we’ve been very happy with the service provided ever since.  Chat’s been all alone out there for over a year now, it deserves some company.  In another blog post coming up I’ll detail how we’re setting up as a read-only disaster recovery location out there, as well as our intention to actively use that while we move datacenters in New York.

P.S. Make sure to RAID your PCIe SSD drives, we’ll put up a post with that story a bit later…

We are going to take a break from our regularly scheduled program to ask a very important question!

I’ve looked a few times, and there are precious few “SysAdmin” conferences. Most of them are very *nix centric. I’m also very interested in Networking, Windows, emerging technologies and all of that fun stuff.

What I have found happens in reality is there are vendor or developer focused conferences with a SysAdmin type track that you can follow. Which when I think about it for a while is probably a good thing. In my mind it allows me to sneak off the beaten path and go check out what the devs are doing. I feel it’s important to not only stay abreast of sysadmin developments, but also keep up with technology in general.

I’ve been to some really good smaller conferences run by vendors that seemed like they would be just one big sales pitch (ok, maybe they were) but they turned out to have some really great breakout sessions run by some really smart people. On the other side of the equation I’ve been to some large conferences that were just awful, where I didn’t pick up anything that wasn’t already general knowledge and everyone just sat there looking bored out of their minds.

So I wanted to run a little informal poll of you, the Great All Knowing Internet.  What conferences to you look forward to going to year after year?  What do you like about them? Also in a more general sense, what are the pros and cons of going to conferences at all? I want to harness this internet power for nothing but good I swear!

Right now Kyle and I have a short list of conferences we would really like to attend:

Also, in addition to conferences you get excited to go to every year, are there any you are especially excited about this year? Let us know over on this Server Fault question: http://meta.serverfault.com/questions/1089/conferences-to-go-to-2011-edition

Please let us know in the comments what conferences you attend every year – maybe we’ll see you there!