This coming Saturday, October 13th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure.  We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of.  We’re planning for mass chaos and lots of fail.

Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon.  Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.

So here’s what we want to happen this weekend:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode
  4. Find problems, squash bugs in our configuration until we’re running smooth
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode
  4. Get beer

What will happen? We don’t know – but we’ll blog here about it.  The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week).  The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.

All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.

Here are a few things we’re afraid (again, why we’re testing):

  • Internal API wonkiness while we’re switching DNS (our sites talk to each other)
  • Bad DNS caching on misconfigured servers/proxies
  • SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
  • 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
  • Oregon doesn’t have enough upstream bandwidth

Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.

«
»
  • Mark Marra

    WHO LET A DEV POST HERE?

    • The sysadmin lines in our company are blurry: http://blog.serverfault.com/2011/05/17/how-many-system-administrators-does-stack-exchange-have/

      • Nick Craver

        No they’re not! Kyle kidnapped me weeks ago and won’t feed me unless I do sysadmin work, save meeeeeeeeeeeeeeeeeeeeeeeeeeeeee

        • Michael Graziano

          It installs the patches on the infrastructure or else it gets the Nimda again!

  • Ward Bush

    And THEN you’ll update Data.SE?

    • Nick Craver

      Data.SE is very soon on the list.  It first has to get a hardware replacement then I’m re-writing the import job since our architecture has tremendously changed since it was last updated.  Once that’s done my goal is to have it on a weekly automated refresh.  So this update is taking longer due to everything else on our plate, but we’re not ignoring it…on the contrary we aim to greatly improve the update frequency there.

  • Michael Graziano

    Re: pager duty, I assume you’re pausing just the part that actually sends out the pages? Events like this are a wonderful way to test your monitoring system’s coverage.

    • Pingdom sends us emails / txts etc, and also tells Pagerduty to call us. So in Pingdom, we pause the “Pagerduty” contact so it doesn’t call the on call person (which happens to be me tomorrow). So we still get the emails and texts, just not the annoying phone calls that I have to acknowledge.

  • Nick Craver

    Note: we’ll have a followup post to this next week after the test…if there are specifics you’re curious about let us know ahead of time and we’ll try a note them during the test.

  • August Winterstein

    Seems ok – the only thing I am wondering is why the LAST step is “Get Beer”.  That should be the first step in case you DO run into horrible problems that take you past the witching hour when the beer stores close.

  • Stuart Leyland

    Thanks for the post, I think it’s awesome how open Stack Exchange is about their architecture and plans, it’s a real difference compared to most companies who are secretive. I look forward to the follow up post about how it went.

  • Marcus

    thanks for blogging about your experience and detailing your setup, looking forward to the follow up. Great to see how it’s done by the pros!

    one small suggestion, that last step (have a beer), wait 1 hour, then have two beers, because sods law says as soon as you have that beer it’ll break 😉

  • John Mckay

    Really interesting, good luck!

  • Andrew Davis

    Definitely shouldn’t be doing this during the RailsRumble. 🙁

  • chriskeeble

    Loving what could be described as a “managing expectations” approach…!! 🙂

    Good luck with it all.

  • James
    • We’re working on it. 🙂 Full write-up to follow.

  • ObsessiveFOSS

     Is it really that hard to show at least a static error page on the frontend server? I’m just getting a useless ASPX error that kindly tells me that I am looking at /questions/unanswered.

    • Nick Craver

      This is a connection error happening extremely early in the pipe, it’s on our list to take a look at.  For the fail back we’ll be disabling the front ends until SQL is up so you’ll see the offline page.

  • ObsessiveFOSS

    There’s an error page now, feel free to delete my spammy comments…

  • Kamyar Paykhan

    Reading how you’re open to the community, leads me to both know more about issues/solutions in the enterprise level and feel more open to contribute. Looking forward to the follow ups.

  • From what I could see just now (site offline for maintenance and back up to full functionality within a few minutes), the test went well… Hope to hear some confirmation. 

  • What step is “restore pager duty”?

    • Nick Craver

      We eventually remember to do this…