Our New York Peer 1 datacenter at 75 Broad is still running on generator power, but as a precaution we decided to failover Stack Overflow, Careers, and the rest of the Stack Exchange network to our secondary datacenter in Oregon last evening. It turned out to be the right call because the refueling trucks can’t get to the facility, so Peer 1 is shutting down all power in about 30 minutes.
We actually recently tested a lot of this, but this is our first time failing over everything at once. So far it is going pretty well, but we have run into a few issues so far:
- An index reorg job kicked off right before failing over. This meant that our SQL replication partners across the country were 40 Gigabytes behind. So Stack Overflow had to remain in read only for about an hour
- Because the status message on our sites is stored in the database, that was readonly, so we couldn’t update it to let everyone know it would be read only for about an hour
- We realized we have to transfer the AD FSMO role forcefully since the NY DCs were shut down, and we don’t know how much fuel is left
- Our backup monitoring system isn’t permitted as an SNMP manager via the Group Policy, so we have to update that
We have some open concerns, and will be keeping a close eye on the following:
- Oregon has some lower end Dell switches, we hope they handle the load. We will be shipping the current 2960S switches to OR once we upgrade our NY switches to the Nexus 5k/2k line in a couple of weeks
- Our load balancers out in OR are a little tight on CPU
- We have 5 web servers in OR instead of 10. However, the combined CPU load of NY on the web tier is ususally 100-200% (Out of 1000%) so I think it will be okay:
However, in the big picture, we have successfully failed over to Oregon! Today is going to feel like taking the sub below its depth rating, you can watch the Das Boot Video to share our feelings.