Just about every introductory class for algorithms teaches you three primary ways to look at algorithm performance:
- Worse Case Scenario
- Average Case Scenario
- Best Case Scenario (This is always followed up by why this isn’t very useful, but they mention it anyways).
You can use this same sort of analysis when thinking about your web performance. I have found that a histogram captures all 3 of these pretty nicely since it is a distribution. The X axis is the response time in milliseconds of a request, and the Y axis is the number of hits that had that response time . Our HAProxy web logs capture the response time of each request from the perspective of the load balancer. This is a nice perspective because it includes pretty much the full stack of a web request that is directly under our control.
We have started to insert our web logs into SQL server. Raw SQL is hard to beat for deep analysis, but I have also started to build a web front end which is particularly useful for generating graphs easily. With a web interface it is easy to filter on certain criteria and you can easily see the distribution for a particular page or a certain client like Googlebot:
A crawler will also be a sort of worst case response time perspective on another level because because crawlers cause more cache misses. Clients will generally perform much differently. For example looking at response time for user agents with “Chrome” in them the response time has a very different shape:
In general, I always find it is best to think about how you are viewing your data, and if it is the best way to summarize what you really care about. Average is useful, but it just isn’t the complete picture.