Working towards failure: Testing this weekend
Nick Craver
This coming Saturday, October 13th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure. We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of. We’re planning for mass chaos and lots of fail.
Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon. Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.
So here’s what we want to happen this weekend:
- Prep (2 hours before the test)
- Shorten DNS TTL down to 5 minutes
- Pause page duty (that’s damn sure going to go off)
- The Test
- Shutdown affected backends in HAProxy (New York)
- Start the DNS swap to Oregon IPs
- Start the SQL 2012 Availability Group failovers to Oregon
- Wait for this to complete before moving forward
- Sanity check sites on the Oregon web tier
- Enable the backends in HAProxy (Oregon)
- Bring the sites out of read-only mode
- Find problems, squash bugs in our configuration until we’re running smooth
- Failing back to New York
- Shut down backends in HAProxy (Oregon)
- Start the DNS swap to New York IPs
- Start the SQL 2012 Availability Group failovers to New York
- Wait for this to complete before moving forward
- Sanity check sites on the New York web tier
- Enable the backends in HAProxy (New York)
- Bring the sites out of read-only mode
- Get beer
What will happen? We don’t know – but we’ll blog here about it. The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week). The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.
All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.
Here are a few things we’re afraid (again, why we’re testing):
- Internal API wonkiness while we’re switching DNS (our sites talk to each other)
- Bad DNS caching on misconfigured servers/proxies
- SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
- 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
- Oregon doesn’t have enough upstream bandwidth
Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.
-
Mark Marra
-
Kyle Brandt
-
Nick Craver
-
Michael Graziano
-
-
-
-
Ward Bush
-
Nick Craver
-
-
Michael Graziano
-
Kyle Brandt
-
-
Nick Craver
-
August Winterstein
-
Stuart Leyland
-
Marcus
-
John Mckay
-
Andrew Davis
-
chriskeeble
-
James
-
Anna Lear
-
-
ObsessiveFOSS
-
Nick Craver
-
-
ObsessiveFOSS
-
Kamyar Paykhan
-
Oded
-
Jed Daniels
-
Nick Craver
-