In order to make upgrades to our network infrastructure, we will be doing maintenance that will involve intermittent downtime tomorrow. The maintenance will start at 2PM UTC on April 14th, 2012 and will continue for several hours.

During this window we will be:

• Replacing our routers and firewalls
• Making some L3 Network design changes

The long term design change will involve changing our WAN and load balancing infrastruture from Active/Passive failover to Active/Active. When this is completed in the following weeks there will be a detailed post explaining our new design!

This Saturday, August 6th starting at 3pm EDT (19:00 UTC) we will be taking our sites offline for maintenance. We are aiming to keep the downtime to about one to four hours. Should we need to extend this window I will post updates on this post.

During this window we are going to be patching our database servers. We are also migrating our servers to a new AD domain so we have everything under a consistent naming structure.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

• When the test was run in UTC time and who ran it
• Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
• Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
• Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.

We’ve been doing a lot of talking about some of the growing pains we have been having with our database server. And how we are going to deal with it. Well we will be moving Stack Overflow’s database to its own hardware this Friday (2011-03-11).

When?

2011-03-11 9:00pm Eastern (2011-03-12 02:00 UTC)

Why?

To Make everything MORE AWESOME

What?

2 new Dell R710’s

• 2x Intel Xeon Processor X5680 3.33 GHz
• 96GB of RAM
• 6 Intel X25-E SSD’s in a RAID 10

What will this mean?

First, we will be putting Stack Overflow into Read Only mode again for the maintenance – just like we did when we went down the Oregon trail. Once the migration is complete, Stack Overflow will be on its own DB pair, and can run free, and the rest of the Stack Exchange sites don’t have to contend with the Megalopolis for database time, making them MORE AWESOME as well.

I will be upgrading the Stack Exchange network’s switching infrastructure on Saturday February, 26th. There should be minimal downtime while I do this, however the sites may be a little slow as I work through the Web Tier. Additionally there will be about a 10-20 minute complete site downtime while I move the DB servers and routers.

What exactly will I be doing? Why, I’ll be removing our current Dell PowerConnect 5448 switches and replacing them with Cisco 2960-S-48TS-L switches.

The plan is to start working on moving the web tier Saturday afternoon and be finished moving all services by early evening. I will post to Blog.Serverfault about one hour before the start of work.

I am going to be doing a bit of DB maintenance tonight at 9PM Eastern. The sites will be down for about an hour while I do a couple of upgrades to our primary database hardware:

1.  Add 32 GB of RAM to the server to bring it up to 96GB of RAM
2. Run Firmware updates for the RAID card
3. Update the RAID Card Driver firmware

The Trilogy and SE sites will be down for about an hour while I do this work. If the outage is going to last longer than that I will update this post.

Also, sorry for not posting this earlier — I’ll try to get these notifications up sooner than an hour before work is scheduled to start in the future.