Before our test last weekend, we posted THE PLAN, as promised here’s a follow-up of how things went:
- Prep (2 hours before the test)
- Shorten DNS TTL down to 5 minutes
- Pause page duty (that’s damn sure going to go off)
- Firewall Oregon redis to prevent mutation (went smooth, late plan addition)
- Slave Oregon redis from the New York master (smooth, late addition)
- The Test
- Shutdown affected backends in HAProxy (New York)
- Start the DNS swap to Oregon IPs
- Start the SQL 2012 Availability Group failovers to Oregon (largest problem)
- Drop redis firewalls in Oregon (went smooth, late plan addition)
- Wait for this to complete before moving forward
- Sanity check sites on the Oregon web tier
- Enable the backends in HAProxy (Oregon)
- Bring the sites out of read-only mode (can be improved)
- Find problems, squash bugs in our configuration until we’re running smooth (went well)
- Firewall New York redis to prevent mutation (broke OpenID)
- Slave New York redis from the Oregon master (smooth, late addition)
- Slave New York redis backup from the New York master/slave (smooth, late addition)
- Oregon went totally offline, twice!
- Failing back to New York
- Shut down backends in HAProxy (Oregon)
- Start the DNS swap to New York IPs
- Start the SQL 2012 Availability Group failovers to New York
- Drop redis firewalls in New York (smooth, late addition)
- Wait for this to complete before moving forward
- Sanity check sites on the New York web tier
- Enable the backends in HAProxy (New York)
- Bring the sites out of read-only mode (wasn’t actually needed)
- Get beer (check, check)
Some of these were late additions to the plan. Having redis be a warm cache once we were up in Oregon meant a few more steps added to the original plan, but well worth it. A cold cache for all sites means stumbling of the servers and slow page loads for the first wave of hits…why have slow pages when they can be fast? The above is a high level plan…the actual one has even more small steps in there, so let’s look at what failed at a high level and some of the smaller details as well.
- Time-wise, the biggest issue was the SQL 2012 always on availability group failover for our SENetwork_AG group; this group contains all of the databases for sites that aren’t stackoverflow.com. While the StackOverflow availability group failed over across the country in seconds, the much larger SENetwork_AG (by database count – that’s what matters in our case) did not. Here’s how that one played out:
- (+0 min): Failover of the SENetwork_AG begins
- (+5 min): After attempting to failover via the SSMS GUI and saw a timeout after 5 minutes
- (+6 min): We attempted to fail it over via script in case this was a tooling timeout in plan
- (+11 min): It’s not a tooling timeout; time to up the default timeouts on the listeners and AG resources in windows
- (+16 min): This had no effect, the 5 minute timeout is somewhere else in the pipe
- (+17 min): As a last ditch effort to get the AG ownership moved to Oregon, I disabled the AG’s dependency on the listener (which we don’t want or need, but have to have)
- (+17.5 min): Success, AG is spinning up
- (+19 min): All databases are back online, SE 2.0 sites are now up
- The second most visible failure was Oregon going completely offline, twice! We have traffic, lots of traffic. This means lots of simultaneous connections to our load balancers, especially when we’re coming up from an outage. This means the default conntrack limits in CentOS 6.3 on our HAProxy load balancers weren’t high enough. We solved this by upping the limit to 1,048,576, matching New York (it turns out we did this weeks ago…fail #2 revealed why it didn’t stick). Later, after another puppet deploy (we have things templated to keep 2 datacenter in sync so…puppet!), the iptables service reloaded. This caused CentOS to unload/reload the iptables module resetting the limit…causing another outage, hoorah. We fixed the limit again and then prevented further reloads – problem solved. This was a good pair of lessons we can apply for when New York load balancers are fully under puppet control.
- The third, lesser-noticed failure was that when we began the redis slaving back to New York to keep that warm cache, we blocked another service using that redis instance: Stack Exchange OpenID. Once we identified this issue we moved it to another instance that isn’t slaved or firewalled as part of a failover. There would be a similar problem when we test OpenID, Careers, etc. failover in a few weeks…so this fix takes care of things for that test as well.
Things that can be better
- When the sites were available (open via HAProxy) but the databases were not yet online, we broadcast a raw error page (YSOD) to users.
- While this can be fixed by not opening the HAProxy backends until the sites are ready, we prefer to at least know what was throwing that error.
- Bringing sites out of read-only mode was more tedious than anticipated
- We have a “disable read-only” button per-site…I’ll be adding a global one as well for situations like this
- Exceptions logging needs some thinking. Our exceptions log to a database that was failed over to Oregon, making it read-only in New York. This meant the services that didn’t failover in New York trying to write to that database had to queue up their exceptions and write them out to the database when it was available for writes again.
- While this was an excellent test of StackExchange.Exceptional’s error queuing in case of database failure…we’d still like better farm-wide visibility during a partial failover.
Overall, we were very happy with how this test went. Most issues were identified and solved quickly, and most of our fears were laid to rest. This has been a long, hard effort by many devs and sysadmins on multiple teams…and we’re not close to being done. This test going very well for the most part has been a very rewarding payoff on our side, we’ll keep you updated as our datacenter move progresses.