In the never ending quest for better performance and response time, perhaps consistency is underrated. In web operations the desire is to make page loads go fast, because performance is a feature, or more simply — people like responsive web pages.

Probably the most common and simplest way to look at response time is to look at the average response time for the primary web request in a page load. Page loads also need to be looked at from the perspective of how they render in a browser (Which is how Google looks at it). This all makes sense, but as you push your load times faster and faster, the savings start to get smaller and smaller. For instance you start reducing the average time of certain requests from something like 1 second to 500 MS, then 500 to 100, then 100 to 80, 50, 30, 15, 10. Eventually these push to limits where you are less likely to feel the difference. It doesn’t mean it is time to stop, but I think in general it makes sense to shift your attention elsewhere for performance gains.

When this happens, a next logical step is often to go for a Content Delivery Network. The CDN can help reduce load times for people who are geographically far from the main servers. We did this back in May for our static content. Then what?

I think a next logical step is to start to shift attention to the consistency of your web server’s response time. By consistency I mean addressing those responses that take a lot longer than your average request time because of the server’s response time. In other words, fixing the outliers. To address these, is quite frankly, a pain. Why?

  • You need a decent sized sample of the response time of every request
  • At hundreds of requests a second, it is hard to pin down the cause of these outliers. It can be code, DB behavior, network blips, etc.

The best way I have come up with to measure this is to pull the response time from our web logs as our load balancer measures it from:

Server response time (HTTP mode only). It’s the time elapsed between the moment the TCP connection was established to the server and the moment the server sent its complete response headers. It purely shows its request processing time, without the network overhead due to the data transmission…

To be honest, I am not sure if this is the best way, but it seems like a reasonable start. I then filtered a sample of log requests so that I was only looking at authenticated users loading a question on that got a 200 response code from the web server (to ease in taking crawlers out of the mix). Then to express the outliers I used a box-and-whisker plot. If you have been wondering just what that image on the right is — this is what you are looking at. You can go read more about it, but in short, the dots are the outliers. When you look at this as a histogram these outliers will be the long tail (which in this case pushes the average to the right of the median).

I have been giving Marc Gravell some data and we were able to pin down a lot of these as collateral damage to a particularly demanding background thread that runs on each server. He is reworking this to help eliminate a lot of these.

I think keeping an eye on these outliers or the long tail of response is going to be tricky, but I also think it plays an important role in the quest for better performance.

  • Can you give a more detailed explanation of the exact nature of the process that caused the damage? Is it just raw CPU crunching that’s causing requests to time out or be delayed or is it a process that hogs database resources specifically?

    I hope this becomes a running series as you work to track down your outliers.

  • This is an interesting-yet-statistically-obvious point – any statistician will tell you that distribution is just as important as any average metric in a given population.

    I was wondering how you’re pulling the data together from HAProxy though? Presumably, you’re using HAProxy to log to a syslog server which then logs to a database, but how are you processing these syslog messages afterwards to get the HTTP timing figures out?

    • @growse: We wrote our own daemon which listens on the syslog port, parses the log entries into fields, and then bulk inserts into SQL server. For the particular image on the right I used R to create the boxplot from that data.

      • You’ve inspired me to crack out perl and do something similar! Got something up and running surprisingly quickly 🙂

  • Jeff Davis

    Very useful post. Most of us stop when the median is fast. But those outliers are when people really get annoyed with sites, and it seems like every site has them.

  • Michagunt24

    raccolta di tatuaggi da scaricare in forma gratuita sopri le offerte di offerte viaggi per le tue vacanze lastminute

  • Aaron

    i wonder if you could shave a few milliseconds by initiating the response to action before the click occurs, say, at the time of the link rollover event – you’d have to have an algorithm to detect pass throughs (mouse in, mouse out on its way to some where else) from landings (mouse in and stop, might as well start doing the work while the brain tells the mulscles in the finger to go ahead and click the button)…. 

  • Pingback: The Importance of Observability - Server Fault Blog()