In the never ending quest for better performance and response time, perhaps consistency is underrated. In web operations the desire is to make page loads go fast, because performance is a feature, or more simply — people like responsive web pages.

Probably the most common and simplest way to look at response time is to look at the average response time for the primary web request in a page load. Page loads also need to be looked at from the perspective of how they render in a browser (Which is how Google looks at it). This all makes sense, but as you push your load times faster and faster, the savings start to get smaller and smaller. For instance you start reducing the average time of certain requests from something like 1 second to 500 MS, then 500 to 100, then 100 to 80, 50, 30, 15, 10. Eventually these push to limits where you are less likely to feel the difference. It doesn’t mean it is time to stop, but I think in general it makes sense to shift your attention elsewhere for performance gains.

When this happens, a next logical step is often to go for a Content Delivery Network. The CDN can help reduce load times for people who are geographically far from the main servers. We did this back in May for our static content. Then what?

I think a next logical step is to start to shift attention to the consistency of your web server’s response time. By consistency I mean addressing those responses that take a lot longer than your average request time because of the server’s response time. In other words, fixing the outliers. To address these, is quite frankly, a pain. Why?

• You need a decent sized sample of the response time of every request
• At hundreds of requests a second, it is hard to pin down the cause of these outliers. It can be code, DB behavior, network blips, etc.

The best way I have come up with to measure this is to pull the response time from our web logs as our load balancer measures it from:

Server response time (HTTP mode only). It’s the time elapsed between the moment the TCP connection was established to the server and the moment the server sent its complete response headers. It purely shows its request processing time, without the network overhead due to the data transmission…

To be honest, I am not sure if this is the best way, but it seems like a reasonable start. I then filtered a sample of log requests so that I was only looking at authenticated users loading a question on stackoverflow.com that got a 200 response code from the web server (to ease in taking crawlers out of the mix). Then to express the outliers I used a box-and-whisker plot. If you have been wondering just what that image on the right is — this is what you are looking at. You can go read more about it, but in short, the dots are the outliers. When you look at this as a histogram these outliers will be the long tail (which in this case pushes the average to the right of the median).

I have been giving Marc Gravell some data and we were able to pin down a lot of these as collateral damage to a particularly demanding background thread that runs on each server. He is reworking this to help eliminate a lot of these.

I think keeping an eye on these outliers or the long tail of response is going to be tricky, but I also think it plays an important role in the quest for better performance.