An update on Sandy

Nick Craver

We’re still up and running in Oregon, and have powered down all services in our New York datacenter at 75 broad.

While we’re not actually running anything out of the datacenter, here’s what’s happening:

After our last update the Peer1 datacenter continues to stay online any way they can.  When Sandy hit, their basement fuel store was contaminated and the pumps necessary to get it up 17 stories to the generator went offline.  Since then employees from several companies hosted there have shown up and continue fueling the generator via bucket brigade.

Our sister company Fog Creek has many hands down there right now, you can follow their status updates here.  Squarespace is in the exact same boat as well, you can see their updates here.

The concern has been fuel supply to carry upstairs, as I’m writing this a truck has pulled up and the barrels are filling.  Several of those companies with people on the ground have been fetching fuel from gas stations for the past few hours.

While this is happening the basement is being pumped out so that the facility can be repaired and grid power restored.  We don’t have an ETA at this time.

We’ll try and post an update in a few hours.

Our New York Peer 1 datacenter at 75 Broad is still running on generator power, but as a precaution we decided to failover Stack Overflow, Careers, and the rest of the Stack Exchange network to our secondary datacenter in Oregon last evening. It turned out to be the right call because the refueling trucks can’t get to the facility, so Peer 1 is shutting down all power in about 30 minutes.

We actually recently tested a lot of this, but this is our first time failing over everything at once. So far it is going pretty well, but we have run into a few issues so far:

  • An index reorg job kicked off right before failing over. This meant that our SQL replication partners across the country were 40 Gigabytes behind. So Stack Overflow had to remain in read only for about an hour
  • Because the status message on our sites is stored in the database, that was readonly, so we couldn’t update it to let everyone know it would be read only for about an hour
  • We realized we have to transfer the AD FSMO role forcefully since the NY DCs were shut down, and we don’t know how much fuel is left
  • Our backup monitoring system isn’t permitted as an SNMP manager via the Group Policy, so we have to update that

We have some open concerns, and will be keeping a close eye on the following:

  • Oregon has some lower end Dell switches, we hope they handle the load. We will be shipping the current 2960S switches to OR once we upgrade our NY switches to the Nexus 5k/2k line in a couple of weeks
  • Our load balancers out in OR are a little tight on CPU
  • We have 5 web servers in OR instead of 10. However, the combined CPU load of NY on the web tier is ususally 100-200% (Out of 1000%) so I think it will be okay:

However, in the big picture, we have successfully failed over to Oregon! Today is going to feel like taking the sub below its depth rating, you can watch the Das Boot Video to share our feelings.

Before our test last weekend, we posted THE PLAN, as promised here’s a follow-up of how things went:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  3. Firewall Oregon redis to prevent mutation (went smooth, late plan addition)
  4. Slave Oregon redis from the New York master (smooth, late addition)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon (largest problem)
  4. Drop redis firewalls in Oregon (went smooth, late plan addition)
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode (can be improved)
  4. Find problems, squash bugs in our configuration until we’re running smooth (went well)
  5. Firewall New York redis to prevent mutation (broke OpenID)
  6. Slave New York redis from the Oregon master (smooth, late addition)
  7. Slave New York redis backup from the New York master/slave (smooth, late addition)
  8. Oregon went totally offline, twice!
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  4. Drop redis firewalls in New York (smooth, late addition)
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode (wasn’t actually needed)
  4. Get beer (check, check)

Some of these were late additions to the plan.  Having redis be a warm cache once we were up in Oregon meant a few more steps added to the original plan, but well worth it.  A cold cache for all sites means stumbling of the servers and slow page loads for the first wave of hits…why have slow pages when they can be fast?  The above is a high level plan…the actual one has even more small steps in there, so let’s look at what failed at a high level and some of the smaller details as well.

Fails

  • Time-wise, the biggest issue was the SQL 2012 always on availability group failover for our SENetwork_AG group; this group contains all of the databases for sites that aren’t stackoverflow.com.  While the StackOverflow availability group failed over across the country in seconds, the much larger SENetwork_AG (by database count – that’s what matters in our case) did not.  Here’s how that one played out:
    • (+0 min): Failover of the SENetwork_AG begins
    • (+5 min): After attempting to failover via the SSMS GUI and saw a timeout after 5 minutes
    • (+6 min): We attempted to fail it over via script in case this was a tooling timeout in plan
    • (+11 min): It’s not a tooling timeout; time to up the default timeouts on the listeners and AG resources in windows
    • (+16 min): This had no effect, the 5 minute timeout is somewhere else in the pipe
    • (+17 min): As a last ditch effort to get the AG ownership moved to Oregon, I disabled the AG’s dependency on the listener (which we don’t want or need, but have to have)
    • (+17.5 min): Success, AG is spinning up
    • (+19 min): All databases are back online, SE 2.0 sites are now up
    We don’t fully understand why this works in New York but not in Oregon…our understanding is it should fail or work equally in both cases, but obviously that’s not correct.  I’ll be following up with Microsoft on what we learned here in hopes it can be improved.
  • The second most visible failure was Oregon going completely offline, twice!  We have traffic, lots of traffic.  This means lots of simultaneous connections to our load balancers, especially when we’re coming up from an outage.  This means the default conntrack limits in CentOS 6.3 on our HAProxy load balancers weren’t high enough.  We solved this by upping the limit to 1,048,576, matching New York (it turns out we did this weeks ago…fail #2 revealed why it didn’t stick). Later, after another puppet deploy (we have things templated to keep 2 datacenter in sync so…puppet!), the iptables service reloaded.  This caused CentOS to unload/reload the iptables module resetting the limit…causing another outage, hoorah.  We fixed the limit again and then prevented further reloads – problem solved.   This was a good pair of lessons we can apply for when New York load balancers are fully under puppet control.
  • The third, lesser-noticed failure was that when we began the redis slaving back to New York to keep that warm cache, we blocked another service using that redis instance: Stack Exchange OpenID.  Once we identified this issue we moved it to another instance that isn’t slaved or firewalled as part of a failover.  There would be a similar problem when we test OpenID, Careers, etc. failover in a few weeks…so this fix takes care of things for that test as well.

Things that can be better

  • When the sites were available (open via HAProxy) but the databases were not yet online, we broadcast a raw error page (YSOD) to users.
    • While this can be fixed by not opening the HAProxy backends until the sites are ready, we prefer to at least know what was throwing that error.
  • Bringing sites out of read-only mode was more tedious than anticipated
    • We have a “disable read-only” button per-site…I’ll be adding a global one as well for situations like this
  • Exceptions logging needs some thinking. Our exceptions log to a database that was failed over to Oregon, making it read-only in New York.  This meant the services that didn’t failover in New York trying to write to that database had to queue up their exceptions and write them out to the database when it was available for writes again.
    • While this was an excellent test of StackExchange.Exceptional’s error queuing in case of database failure…we’d still like better farm-wide visibility during a partial failover.

Overall, we were very happy with how this test went.  Most issues were identified and solved quickly, and most of our fears were laid to rest.  This has been a long, hard effort by many devs and sysadmins on multiple teams…and we’re not close to being done.  This test going very well for the most part has been a very rewarding payoff on our side, we’ll keep you updated as our datacenter move progresses.

This coming Saturday, October 13th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure.  We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of.  We’re planning for mass chaos and lots of fail.

Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon.  Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.

So here’s what we want to happen this weekend:

  • Prep (2 hours before the test)
  1. Shorten DNS TTL down to 5 minutes
  2. Pause page duty (that’s damn sure going to go off)
  • The Test
  1. Shutdown affected backends in HAProxy (New York)
  2. Start the DNS swap to Oregon IPs
  3. Start the SQL 2012 Availability Group failovers to Oregon
  • Wait for this to complete before moving forward
  1. Sanity check sites on the Oregon web tier
  2. Enable the backends in HAProxy (Oregon)
  3. Bring the sites out of read-only mode
  4. Find problems, squash bugs in our configuration until we’re running smooth
  • Failing back to New York
  1. Shut down backends in HAProxy (Oregon)
  2. Start the DNS swap to New York IPs
  3. Start the SQL 2012 Availability Group failovers to New York
  • Wait for this to complete before moving forward
  1. Sanity check sites on the New York web tier
  2. Enable the backends in HAProxy (New York)
  3. Bring the sites out of read-only mode
  4. Get beer

What will happen? We don’t know – but we’ll blog here about it.  The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week).  The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.

All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.

Here are a few things we’re afraid (again, why we’re testing):

  • Internal API wonkiness while we’re switching DNS (our sites talk to each other)
  • Bad DNS caching on misconfigured servers/proxies
  • SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
  • 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
  • Oregon doesn’t have enough upstream bandwidth

Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.