Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.


We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

  • Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
  • Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
  • Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
  • Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

  • There are constraints of what is actually available to buy, and how much you can actually know what you are getting
  • Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.


When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

  • Gave us time to rectify some mistakes
  • Gave us confidence in our secondary datacenter since we failed over to it before the storm
  • Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.


Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

StackExchange welcomes our newest system administrator, Steven Murawski!

Steven Murawski Steven followed a bit of an eclectic path to systems administration, having started out operating a garden center, floral design and landscape business. After eight years he decided to move into law enforcement; he went to school for Police Science and worked at a local police department as a clerk, dispatcher and auxiliary officer. While he enjoyed shooting various firearms and driving at high rates of speed, he also began working with the police department’s IT systems, and slowly fell in love with systems administration. He became a member of LOPSA, Microsoft MVP, and continues to lead two local user groups (the Greater Milwaukee IT Pro User Community and Greater Milwaukee Script Club). Steven combined his love of systems administration with public speaking by presenting technology talks at various technology conferences and user groups.

Today Steven distracts himself from technology by playing with his eight month old son, puttering around with home-improvement projects and hurling pieces of lead through paper targets, although he is seriously considering once again picking up the mantle of host for a systems administration and Powershell podcast!

About a  year ago, I wrote a post about the basics of getting a job in Information Technology.  Unfortunately, that post (linked here) did not cover much relevant material when it comes to putting your resume in and getting an interview.  This blog post has been in the making for a long time, however the major impetus came when a serverfault regular tossed me a message over the weekend asking for suggestions on how to answer “The Salary Question.” I address this below, but first I will cover what I think is the single most important thing to remember in IT when it comes to getting hired.

“It’s not what you know, it’s who you know!”

I have heard many people bemoan the fact that it is near-impossible to get an interview for tech jobs in this market.  Their stories are generally the same, listed below:

  • See there is a position at Awesome Company, LLC.
  • See that the resume is asking for every technology buzzword imaginable
  • Asks for 10 years experience with Windows 8
  • Decide to send in your resume for it, even though you only know 3 or 4 of the 15 buzzwords they’ve specified.
  • Become depressed when you don’t get a callback.

This is not an unusual turn of events.  The reason behind this oftentimes is that HR is putting together a job description and they’re (optionally!) asking the team that’s hiring for “some technologies they should look for.”  Oftentimes that hiring team says “Well, if they know ABC, XYZ, TLA, ETLA, or Java, it’d be a plus.”   This turns into “applicant must have X years experience in ABC, XYZ, TLA, ETLA and Java.”  HR just screwed not only your chances of getting hired, but the hiring team’s chance at getting a potentially good candidate.

The thing that is tough for most people to understand is that all of the above is what I like to call an “HR Trap.”  It’s there to trap your resume and never get it to the hiring team for consideration.  So, as problem solving IT thinkers, we need to think around this problem.

Do you have any friends that work at Awesome Company, LLC?  Are they friends with the IT guys, per chance?  Ask your friends if they’d be willing to hand your resume to the team that’s doing the hiring along with a recommendation.  (NOTE: Don’t sound needy!  It’s a good way to sour friendships.)  Some people advocate saying “Hi James, I noticed your company is hiring for a position that I’d be great at, and I’d really love to get an interview.  Would you put my resume on Tim The IT Director’s desk for me and possibly put in a good word or two on my behalf?”  This can really work.

Network, Network, Network!

There’s another thing we can do to increase our chances of getting hired in the IT community: go meet people!  This is called “Networking” to the sales world and is critical in getting their job done.  I do know that a lot of us consider ourselves antisocial, but realistically, if all of us “antisocial” types get together, we generally find things to talk about.  Don’t be afraid to approach people, please!  This process only works when you are the one taking initiative.  Here are some options for networking:

  • Conferences (LISA, LOPSA-East (nee PICC), Cascadia, Velocity, many others) are great places to network with other people and rub shoulders with the giants of our industry.  For instance, Vint Cerf will be delivering the keynote at LISA this year.  If you ever wanted to meet him, if you were at LISA you could just go up and say hi.
  • Local user groups (Like LOPSA presentations, Linux User Groups, Windows User Groups, Tech Meetups) help a great deal as well, and for most of us without a big pocketbook, they’re much cheaper to attend than a conference.  You still get an opportunity to meet a gaggle of new people, and that’s helpful when you’re looking for a job.  I know that LOPSA especially, at least the chapter that I belong to, asks if anyone has job openings they’d like to talk about.  If someone’s asking for something you can do, go talk to them!  You’ve just instantly gotten ahead of the game by meeting the possible stakeholder(s) before HR has had the chance to accidentally delete your e-mailed resume.

Show the world how awesome you are.

An oft overlooked part of the hiring process is the inevitable google search that your potential employer is going to do.  What kind of stuff are they going to find?  Are your Facebook photos from your buddy’s beer pong tournament open to the public?  This is what could be called a “negative public artifact.”  Some tech companies would find it funny, but stuffier business types would consider that an immediate turn off.  On the other hand, if you have a blog full of insightful comments on the industry and diary entries of your (mis)adventures in technology, you show that you’re not simply a piece of paper with a lot of certifications, but someone who actually produces good things.  This is where being a member on a Stack Exchange site is very helpful.  If you mention you’re a serverfault user on your resume and provide the link to your profile, your potential employer has one place to look for all of your questions and answers to judge how well you know your stuff.

Fun With Resumes

I know a lot of us are holding onto the same resume format we had when we started, and are just adding fields as necessary; it’s not always a smart idea.  The job market and what people are looking for on resumes has changed since some of us started looking for jobs.  The emphasis is shifting from “show your experience” to “show how you’re awesome;” if you want your resume to get a second glance, it has to clearly and concisely illustrate why you’re a slam dunk.

1972-1977  Sanford and Son, Salvage

  • Drove a salvage truck
  • Excellent problem solver

If your resume has sections like the above, you are boring the heck out of the person reading your resume.  Sure, it shows what you literally did in that job, but it doesn’t convey what you’re proud of.  Did you make the salvage equivilent of the Kessel run in less than 12 parsecs?  How is the potential employer going to know about this if you didn’t say so?  Lets try this again:

Sanford and Son, Salvage (1972-1977)

  • Frequently noted as having the best one-liners in the company
  • Kept a 1952 Ford Pickup in running order much longer than previously considered humanly possible
  • Voted the fastest salvager on the east side after completing the Kessel Run in under 12 parsecs! (City record!)

This is not unusual advice.  I’m sure many of us learned it in school when they talked about writing resumes.  At the very least, we’ve spiced up an otherwise dull list of abilities/accomplishments.  Why stop there, we can still do better:

Sanford and Son, Salvage (1972-1977)


  • Ran the Los Angeles Kessel Run in under 12 blocks, beating the standing record of 16 blocks. (1972)


  • In Trash We Trust Award – 1973
  • Salvage Monthly’s “Top of the Heap” – 1975

Skills Utilized

  • Advanced engine mechanics and collision repair
  • Navigation of South-Central Los Angeles by automobile and commercial trucks
  • Lifting heavy loads

If we look at the above, two things are important to note.  First, it has easy to read sections.  Second, it has superfluous space taken up by sections.  If you were to put the whole body of your experience down in this format, you’d have a 4-5 page resume.  Some places might like this, but the common knowledge is 2-3 pages, max.  Without some massaging, your long-winded resume will be too troublesome to read.  Why?  Oftentimes we’re handed a stack of resumes and after a while, it all gets so repetitive that you confuse which applicant had which experience.  The key to resolving this is massaging your work experience to be relevant to the job that you’re applying for.

Tailor that resume, chief

I know that it’s hard work to tailor resumes for every position, but at the very least you should have a few tailored resumes in the areas that you are most proficient.  For example, I have three separate resumes that I give out if I’m applying for a developer, IT, or management position.  I’ve done all three jobs, so I have three tailored sets of job experience that subtly (or perhaps not-so-subtly) emphasize those roles in the full body of my experience.  Sometimes I leave a job out that isn’t relevant to the position, but I’ve found some interviewers are taken aback when you mention you left out some job experience.  Why would  you care if I slung lattes at a coffee bar while in high school, I ask?

Cover letters are mandatory

You might have heard people say that cover letters are dying out in this age of digital resumes.  They’re lying so they can take your jobs.  Every single resume you sent out needs a cover letter, and it needs to be customized to the company you want to work for.  I know that this is a tall order for some, but if you’re a prudent applicant you’ve already studied the company you’re applying for and know what they do or provide.  Why not explain to the person who’s going to hire you why they should waste their day interviewing you?  The cover letter is your “First Impression” on paper.  It helps to show how eloquently you convey yourself on paper, and it’s  pretty important in tehnology jobs because a lot of our jobs involve some level of technical writing.  Ignore this advice at your peril.

Please provide your starting salary…

Yeah, if you want to get priced out of the market.  The answer to that question should always be “we can negotiate that after the interview when you’re sure I’ll be a great fit!”  If they press, try to dodge as much as possible.  The salary question allows them to automatically throw out resumes.  If you come in low, they think you’re not in touch with the job requirements or the job market.  If you come in high, they’ll think you’re either presumptuous or literally are too experienced and therefore too expensive to hire.

Your resume will give them an accurate idea of how much they should be paying you just by the virtue of how much experience you’re showing.  If you’d like more details on this, check out this year’s Robert Half Salary Guide.  This gives an accurate starting point to figure out how much money you should be asking for or how much they should be paying you.  However, don’t tell them this number (or your imaginary What-Am-I-Worth-A-Year number) until after they’ve decided they want to hire you.

Closing thoughts

I don’t want to end this on a somber note, but please keep in mind that if you’re trying to get interviews at companies where you have no inroads or networking contacts, please be ready to get a callback from 1 out of every 20 or more resumes you send.  It is a cut-throat job market and there are hundreds if not thousands of people applying for the same positions you are, but they’re better at bullshitting on resumes than you are.  It’s tough to get a job without having friends in the industry already.  Don’t get discouraged!  Do what I advise above, especially on the socializing front, and you’ll be much better off.

Appendix 1: What a well-formed Systems Administrator job reference might look like

I was asked to provide an actual SysAdmin-focused job description to help some people along.  Below is a completely fictitious entry that I’ve generated off the top of my head.

Senior Systems Administrator, ImaginedCompany, LLC (2008-Present)

Relevant Technologies

  • Advanced Active Directory provisioning and maintenance for a 100,000 user company
  • Deployment automation via Windows Deployment Services in Windows and KickStart in Linux
  • vSphere setup, troubleshooting and administration with multiple Datacenters and Hosts
  • Deep Telephony – PRI, TDM, FXO/FXS setup and maintenance
  • Configuration Automation via Puppet and Systems Center
  • Revision control with Mercurial, Git and Subversion


  • Independently developed the Windows Deployment Services installation for ImaginedCompany, decreasing provisioning time from hours to minutes.
  • Converted legacy servers from bare-metal hardware to vmware, saving the company thousands of dollars a year in electricity
  • Quickly reimplemented a PBX solution when the legacy telephone system hardware failed, enabling the company to continue to make money until a permanent solution was in place.
  • Led-by-example with new project to use source and configuration management by way of Mercurial and Puppet/Systems Center


  • 2008 – LOPSA Most Active Mentor
  • 2009 – Golden Baseball Bat recipient for best client-facing customer service.
  • 2011 – Employee of the Year

An update on Sandy

Nick Craver

We’re still up and running in Oregon, and have powered down all services in our New York datacenter at 75 broad.

While we’re not actually running anything out of the datacenter, here’s what’s happening:

After our last update the Peer1 datacenter continues to stay online any way they can.  When Sandy hit, their basement fuel store was contaminated and the pumps necessary to get it up 17 stories to the generator went offline.  Since then employees from several companies hosted there have shown up and continue fueling the generator via bucket brigade.

Our sister company Fog Creek has many hands down there right now, you can follow their status updates here.  Squarespace is in the exact same boat as well, you can see their updates here.

The concern has been fuel supply to carry upstairs, as I’m writing this a truck has pulled up and the barrels are filling.  Several of those companies with people on the ground have been fetching fuel from gas stations for the past few hours.

While this is happening the basement is being pumped out so that the facility can be repaired and grid power restored.  We don’t have an ETA at this time.

We’ll try and post an update in a few hours.

Our New York Peer 1 datacenter at 75 Broad is still running on generator power, but as a precaution we decided to failover Stack Overflow, Careers, and the rest of the Stack Exchange network to our secondary datacenter in Oregon last evening. It turned out to be the right call because the refueling trucks can’t get to the facility, so Peer 1 is shutting down all power in about 30 minutes.

We actually recently tested a lot of this, but this is our first time failing over everything at once. So far it is going pretty well, but we have run into a few issues so far:

  • An index reorg job kicked off right before failing over. This meant that our SQL replication partners across the country were 40 Gigabytes behind. So Stack Overflow had to remain in read only for about an hour
  • Because the status message on our sites is stored in the database, that was readonly, so we couldn’t update it to let everyone know it would be read only for about an hour
  • We realized we have to transfer the AD FSMO role forcefully since the NY DCs were shut down, and we don’t know how much fuel is left
  • Our backup monitoring system isn’t permitted as an SNMP manager via the Group Policy, so we have to update that

We have some open concerns, and will be keeping a close eye on the following:

  • Oregon has some lower end Dell switches, we hope they handle the load. We will be shipping the current 2960S switches to OR once we upgrade our NY switches to the Nexus 5k/2k line in a couple of weeks
  • Our load balancers out in OR are a little tight on CPU
  • We have 5 web servers in OR instead of 10. However, the combined CPU load of NY on the web tier is ususally 100-200% (Out of 1000%) so I think it will be okay:

However, in the big picture, we have successfully failed over to Oregon! Today is going to feel like taking the sub below its depth rating, you can watch the Das Boot Video to share our feelings.

Before our test last weekend, we posted THE PLAN, as promised here’s a follow-up of how things went:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  3. Firewall Oregon redis to prevent mutation (went smooth, late plan addition)
  4. Slave Oregon redis from the New York master (smooth, late addition)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon (largest problem)
  4. Drop redis firewalls in Oregon (went smooth, late plan addition)
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode (can be improved)
  4. Find problems, squash bugs in our configuration until we’re running smooth (went well)
  5. Firewall New York redis to prevent mutation (broke OpenID)
  6. Slave New York redis from the Oregon master (smooth, late addition)
  7. Slave New York redis backup from the New York master/slave (smooth, late addition)
  8. Oregon went totally offline, twice!
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  4. Drop redis firewalls in New York (smooth, late addition)
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode (wasn’t actually needed)
  4. Get beer (check, check)

Some of these were late additions to the plan.  Having redis be a warm cache once we were up in Oregon meant a few more steps added to the original plan, but well worth it.  A cold cache for all sites means stumbling of the servers and slow page loads for the first wave of hits…why have slow pages when they can be fast?  The above is a high level plan…the actual one has even more small steps in there, so let’s look at what failed at a high level and some of the smaller details as well.


  • Time-wise, the biggest issue was the SQL 2012 always on availability group failover for our SENetwork_AG group; this group contains all of the databases for sites that aren’t  While the StackOverflow availability group failed over across the country in seconds, the much larger SENetwork_AG (by database count – that’s what matters in our case) did not.  Here’s how that one played out:
    • (+0 min): Failover of the SENetwork_AG begins
    • (+5 min): After attempting to failover via the SSMS GUI and saw a timeout after 5 minutes
    • (+6 min): We attempted to fail it over via script in case this was a tooling timeout in plan
    • (+11 min): It’s not a tooling timeout; time to up the default timeouts on the listeners and AG resources in windows
    • (+16 min): This had no effect, the 5 minute timeout is somewhere else in the pipe
    • (+17 min): As a last ditch effort to get the AG ownership moved to Oregon, I disabled the AG’s dependency on the listener (which we don’t want or need, but have to have)
    • (+17.5 min): Success, AG is spinning up
    • (+19 min): All databases are back online, SE 2.0 sites are now up
    We don’t fully understand why this works in New York but not in Oregon…our understanding is it should fail or work equally in both cases, but obviously that’s not correct.  I’ll be following up with Microsoft on what we learned here in hopes it can be improved.
  • The second most visible failure was Oregon going completely offline, twice!  We have traffic, lots of traffic.  This means lots of simultaneous connections to our load balancers, especially when we’re coming up from an outage.  This means the default conntrack limits in CentOS 6.3 on our HAProxy load balancers weren’t high enough.  We solved this by upping the limit to 1,048,576, matching New York (it turns out we did this weeks ago…fail #2 revealed why it didn’t stick). Later, after another puppet deploy (we have things templated to keep 2 datacenter in sync so…puppet!), the iptables service reloaded.  This caused CentOS to unload/reload the iptables module resetting the limit…causing another outage, hoorah.  We fixed the limit again and then prevented further reloads – problem solved.   This was a good pair of lessons we can apply for when New York load balancers are fully under puppet control.
  • The third, lesser-noticed failure was that when we began the redis slaving back to New York to keep that warm cache, we blocked another service using that redis instance: Stack Exchange OpenID.  Once we identified this issue we moved it to another instance that isn’t slaved or firewalled as part of a failover.  There would be a similar problem when we test OpenID, Careers, etc. failover in a few weeks…so this fix takes care of things for that test as well.

Things that can be better

  • When the sites were available (open via HAProxy) but the databases were not yet online, we broadcast a raw error page (YSOD) to users.
    • While this can be fixed by not opening the HAProxy backends until the sites are ready, we prefer to at least know what was throwing that error.
  • Bringing sites out of read-only mode was more tedious than anticipated
    • We have a “disable read-only” button per-site…I’ll be adding a global one as well for situations like this
  • Exceptions logging needs some thinking. Our exceptions log to a database that was failed over to Oregon, making it read-only in New York.  This meant the services that didn’t failover in New York trying to write to that database had to queue up their exceptions and write them out to the database when it was available for writes again.
    • While this was an excellent test of StackExchange.Exceptional’s error queuing in case of database failure…we’d still like better farm-wide visibility during a partial failover.

Overall, we were very happy with how this test went.  Most issues were identified and solved quickly, and most of our fears were laid to rest.  This has been a long, hard effort by many devs and sysadmins on multiple teams…and we’re not close to being done.  This test going very well for the most part has been a very rewarding payoff on our side, we’ll keep you updated as our datacenter move progresses.

This coming Saturday, October 13th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure.  We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of.  We’re planning for mass chaos and lots of fail.

Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon.  Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.

So here’s what we want to happen this weekend:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode
  4. Find problems, squash bugs in our configuration until we’re running smooth
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode
  4. Get beer

What will happen? We don’t know – but we’ll blog here about it.  The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week).  The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.

All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.

Here are a few things we’re afraid (again, why we’re testing):

  • Internal API wonkiness while we’re switching DNS (our sites talk to each other)
  • Bad DNS caching on misconfigured servers/proxies
  • SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
  • 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
  • Oregon doesn’t have enough upstream bandwidth

Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.

It is pretty hard to get a bunch of system administrators together for any period of time and not have a conversation about command line for managing things. In my experience, command line always wins the debate for any medium to large installation. Microsoft caught on to this and has introduced powershell. From my own experience and people I have talked to, powershell adoption has been slow. Like powershell or dislike it, it is an interesting take on command line with its object oriented versus text-based approach. It is also quite powerful, so why the slow adoption?

In my mind it isn’t a problem with powershell itself, but rather a result of human nature, culture, and the Windows ecosystem. The problem lies in workflow, and I think the best way to illustrate this is to contrast Windows versus Linux administration. When managing Linux systems, scripting something flows naturally from the experience of trying things out in the first place. For example, lets say someone with some Linux experience (but not an expert) is building a piece of software and installing it on Linux for the very first time. The steps usually involve something like:

  1. Run the traditional, configure, make, make install from the command line. See what libraries or binaries are missing that are required to build, from the errors. Install them (often via yum or apt), keep doing this until it works.
  2. Possibly add some users to the system, adjust permissions etc with commands like useradd, chown, chmod.
  3. Edit some text files
  4. Set it to start at boot with commands like chkconfig or update-rc.d

As you do this more and more, you learn the patterns of what fails, and get the urge to automate it. Although things like puppet are the end goal these days, the next normal step is to write shell scripts to do this. This is the lynchpin of the whole experience, a shell script is just a little flow control, error handling, and a few variables on top of what you were already doing. With the Linux experience, scripting flows naturally as a next step in your experience. It is a gradual learning experience, and your previous experience of doing things manually taught you much of what you need to know to script it — what goes wrong, what to check, how to do everything you need to do. Even the first time tutorials on the web are almost always command line. From the second you start learning how to administrate a Linux box, you are learning how to script.

Unless you are extremely disciplined or have been taught Windows administration recently by a “benevolent” dictator, your first steps in Windows administration are via the GUI and Wizards. Don’t get me wrong, for many things this gets the job done faster and with far more ease. The generally accepted problem with the GUI though is that it doesn’t scale. When you want to scale in management you often turn to scripts (GPOs are very useful, but you are generally out of luck if what you want isn’t predefined.) With Windows, scripting these tasks is generally a complete departure from your previous experience. Scripting doesn’t flow naturally from previous experience, and scripting something is scary because of this. This means the windows ecosystem, due to human nature, is far less likely to produce administrators with scripting experience. Therefore, powershell adoption is logically slower. This also means administrators are less likely to be testing powershell and pressuring each other and Microsoft to improve it.

The cure to this workflow to me seems to be Windows Core since changing human nature isn’t easy. With Windows core, even though you can use the Management Console, you really have to start doing many things via the command line and powershell. With use of these tools, just as with the shell in Linux, scripting will flow naturally. The Catch-22 however though is that since adoption is slow, everything you need might not be available via command line options. At my company we haven’t used Windows core, but I’m hoping with Server 2012 it is time for Core, does anyone with more experience know if it is time for this shift?

The move to SQL 2012

Nick Craver

Sorry this blog has been a bit quiet lately, we’ve been very busy making some big changes behind the scenes.  So what are we up to?  Let’s start with just the SQL infrastructure moves, here’s a list of servers in play as they started out:

  • NY-DB01 – SQL2008 R2 Hosts all sites except Stack Overflow and the Sites DB
  • NY-DB02 – SQL2008 R2 Daily backups restored from NY-DB01, and the Dev DBs
  • NY-DB03 – SQL2008 R2 StackOverflow’s DB
  • NY-DB04 – SQL 2008 R2 StackOverflow 5 minute behind hot spare in restore mode
  • OR-DB01 – SQL2008 R2 Chat’s DBs
  • OR-DB02 – SQL2008 R2 SEDE, Internal SEDE and Chat Dev DBs
  • New Dell R710 w/ 2x OS Drives + 6x Data 300GB Intel 320 SSDs in RAID10 and 96GB RAM
  • 3xNew Dell R720 w/ 2x OS Drives +  12x Data 200GB Intel 710 SSDs in RAID10 and 384GB RAM

First we set up the first SQL2012 Cluster with those new R720 machines. The new R720s are identical; they became NY-SQL01, NY-SQL02 and OR-SQL01:

  • NY-SQL01
    • Primary: Sites DB, Stack Overflow
    • Backups: Sites DB & StackOverflow Full and Transaction Logs -> NY
  • NY-SQL02
    • Replica: Sites DB, Stack Overflow, Chat DBs
    • Backups: Chat DBs Full -> NY
  • OR-SQL01
    • Primary: Chat DBs
    • Replica: Sites DB, Stack Overflow
    • Backups: Sites SB, StackOverflow & Chat DBs Full -> OR

For this we have 2 new availability groups, StackOverflow_AG and Chat_AG. The primary server for StackOverflow_AG is NY-SQL01 replicated to a secondary in the same data center (NY-SQL02) and across the country to Oregon (OR-SQL01).  The Chat_AG is only on 2 servers: the OR-SQL01 primary (chat is hosted in Oregon) and the replica NY-SQL02. The reason chat is only on 2 servers is because SQL2012 availability groups do not have the ability to distinguish between sites and replicate that way…so it would send the same transaction stream across the country twice to replicate to the NY servers, rather than echoing the transactions through one to the other…this is an unnecessary use of bandwidth we feel.

The StackOverflow and Sites DB portion of the first cluster was completed on the 2012-08-11 maintenance window; chat will be completed on 2012-08-18 (part of chat has moved, we want to give it a week to observe any problems).  Now what happens in the following week? We need to shuffle some hardware around.

With the StackOverflow DB moved off of the NY-DB03 and NY-DB04 pair, they’re ready to be re-tasked.  Currently these servers are identical Dell R710s with 288GB of RAM, 2x OS Drives in a RAID 1 and 6x 200GB Intel 710 SSDs in RAID10.  These boxes get re-tasked to be NY-SQL03 and NY-SQL04. Joining them in this second SQl2012 cluster is OR-SQL02, that new Dell R710 above.  Here’s a breakdown:

  • NY-SQL03
    • Primary: All Stack Exchange 2.0 Sites other than SO
    • Backups: SE 2.0s Full + [diff of trans] -> NY
  • NY-SQL04
    • Primary: SE 2.0 & SO Dev DBs
    • Replica: All Stack Exchange 2.0 Sites other than SO
  • OR-SQL02
    • Primary: Chat Dev DBs
    • Replica: All Stack Exchange 2.0 Sites other than SO
    • SE 2.0s Full + [diff of trans] -> OR

Now we’ve freed up the NY-DB01 and NY-DB02 boxes, they’ll also be nuked, get some new drives and be re-tasked for some other purposes (for example, one goes to Oregon to be the HAProxy traffic log out there).

For the miscellaneous bits, OR-DB01 will be freed up after the move to OR-SQL01 and OR-SQL02 of the chat DBs.   We’ll then take OR-DB01 and install 2012 re-tasking it to host the databases.  It has double the memory of the current server and should provide a nice boost to performance there.


Why? What does all this moving get us?  Well it turns out SQL 2012 Always on Availability Groups give us quite a bit.  Here are the big ones for our architecture:

  • Near real-time replicas of every production database, ready to go
  • No more copying backups across to the offsite datacenter for redundancy
  • We can read from the replicas, eliminating the need for an entire server and allows us to spread the read load out (e.g. API can point at a replica)
  • A backup DR location is now doable

First, we can have very near real-time hot spares for all production databases (previously, we had up to 8 hours data loss between differentials).  Second, we don’t need to do these wasteful copies of databases across the country purely for backup purposes…we have a nearly-in-sync replica across the country we can do speedy local backups from.  That’s a huge cross-country VPN bandwidth savings as an added bonus.  Third, we can spread the read load out across multiple servers (and we can add another 2 more to either of these availability groups if needed).  Performance-wise, we don’t even have a need for any read load spreading, but it’s very nice to have as an option. Now for the last one: a DR site.

PEAK Internet in Oregon is where Stack Overflow began on a single server, and we’ve been very happy with the service provided ever since.  Chat’s been all alone out there for over a year now, it deserves some company.  In another blog post coming up I’ll detail how we’re setting up as a read-only disaster recovery location out there, as well as our intention to actively use that while we move datacenters in New York.

P.S. Make sure to RAID your PCIe SSD drives, we’ll put up a post with that story a bit later…

OODA for Sysadmins

Kyle Brandt

Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.

OODA in Practice

As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.

Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.

Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.

The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.

This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.

Common Problems

This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.

When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.

Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.

If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.

OODA Done Right

Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.

With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.

Why it Matters

If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.