When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

• When the test was run in UTC time and who ran it
• Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
• Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
• Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.