Recently there was a question on the Programmers site asking “Why don’t all companies buy developers the best hardware?” This is actually a very interesting question and there was a good deal of discourse on the topic, as there always is when you get a group of professionals talking about the tools of their trade.

We here at Stack Exchange pride ourselves on not letting the technology get in the way of our devs. We want them to be able to do their job – which is writing code, and doing it well – with minimal hassle. Now, I have to take a brief moment to make a little bit of a note some of the things we do do not scale.

The philosophy that is maintained when getting a developer a new machine is a pretty simple one:

> If a dev is constantly struggling with their machine they aren’t getting work done, they aren’t happy, and they are producing bad code. All of these things are more expensive than a nice developer machine

As a sysadmin – and hence the guy in charge of getting the devs what they need you find out really quickly that they know what they want for the most part, and for those that you don’t ask if you give them a powerful machine with a lot of screen real estate they are generally very happy.

Since I mentioned earlier that there are some things we do here that aren’t all that scalable – in that they work for a company with 20 devs but not 100 i’ll split them up.

The Scalable

The base config that I get new devs in the NY office is:

  • Dell Optiplex 980 class tower
  • Max out the RAM
  • Best i7 Processor that I can get in it
  • SSD primary drive
  • 7.2k large (500GB – 1TB) Secondary Drive
  • 30″ Primary Monitor
  • 20″ Secondary monitor (turned sideways)

Every dev picks their own keyboard – we may give them the crappy one that comes with the machine to get them up and running, but they can request any keyboard/mouse combo they want (and … i do mean any).

The not so scalable

  • For our remote devs we get workstation replacement class laptops. Basically I go out and find the most powerful laptop I can get at the time they start, as well as a 30″ monitor to go with it. Not Scalable.
  • At least one dev in the NY office has a fully customized hand built machine with 2 30″ monitors. VERY not scalable.
  • Whatever they want – within reason

Basically, it boils down to get them what they need to get their job done. One of the biggest challenge to some people is that they do not have management’s buy-in to get the Devs what they need. I find that this is a sad state of affairs, but I have no real advice to offer – especially since I work somewhere that management has mandated great dev machines.

In the never ending quest for better performance and response time, perhaps consistency is underrated. In web operations the desire is to make page loads go fast, because performance is a feature, or more simply — people like responsive web pages.

Probably the most common and simplest way to look at response time is to look at the average response time for the primary web request in a page load. Page loads also need to be looked at from the perspective of how they render in a browser (Which is how Google looks at it). This all makes sense, but as you push your load times faster and faster, the savings start to get smaller and smaller. For instance you start reducing the average time of certain requests from something like 1 second to 500 MS, then 500 to 100, then 100 to 80, 50, 30, 15, 10. Eventually these push to limits where you are less likely to feel the difference. It doesn’t mean it is time to stop, but I think in general it makes sense to shift your attention elsewhere for performance gains.

When this happens, a next logical step is often to go for a Content Delivery Network. The CDN can help reduce load times for people who are geographically far from the main servers. We did this back in May for our static content. Then what?

I think a next logical step is to start to shift attention to the consistency of your web server’s response time. By consistency I mean addressing those responses that take a lot longer than your average request time because of the server’s response time. In other words, fixing the outliers. To address these, is quite frankly, a pain. Why?

  • You need a decent sized sample of the response time of every request
  • At hundreds of requests a second, it is hard to pin down the cause of these outliers. It can be code, DB behavior, network blips, etc.

The best way I have come up with to measure this is to pull the response time from our web logs as our load balancer measures it from:

Server response time (HTTP mode only). It’s the time elapsed between the moment the TCP connection was established to the server and the moment the server sent its complete response headers. It purely shows its request processing time, without the network overhead due to the data transmission…

To be honest, I am not sure if this is the best way, but it seems like a reasonable start. I then filtered a sample of log requests so that I was only looking at authenticated users loading a question on stackoverflow.com that got a 200 response code from the web server (to ease in taking crawlers out of the mix). Then to express the outliers I used a box-and-whisker plot. If you have been wondering just what that image on the right is — this is what you are looking at. You can go read more about it, but in short, the dots are the outliers. When you look at this as a histogram these outliers will be the long tail (which in this case pushes the average to the right of the median).

I have been giving Marc Gravell some data and we were able to pin down a lot of these as collateral damage to a particularly demanding background thread that runs on each server. He is reworking this to help eliminate a lot of these.

I think keeping an eye on these outliers or the long tail of response is going to be tricky, but I also think it plays an important role in the quest for better performance.

The Three Perspectives

In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:

  • Micro: “Ground Level”
  • Meso: “Day to Day”
  • Macro: “Seasonal”

These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.

From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.

I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.

Ground Level Monitoring

“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:

  • Perfmon
  • Sar
  • Wireshark / TCPDump
  • Web Logs (or other detailed logs)
  • SQL Server Profiler

However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.

In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.

The Attributes of Micro Data

The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:

  • Different sets of tools to process the data are needed
  • Generally samples of high resolution data and not complete sets are used
  • To correlate all of this, you will need to do some work because it will be different for every environment

Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.

One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).

Case Study: Web Logs and Perfmon Data

Data Analysis Platforms

Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.

Getting the Data

For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.

Exploring the Data with R

I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):

Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.

The Difference?

If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.

What is Next?

I don’t have a high resolution system deployed yet. So I need work on and think about:

  • Scheduling Perfmon
  • How to get the most representative samples
  • Scheduling TCPDump and getting that data into R
  • Scheduling Sar on the Linux boxes and getting that into R
  • Automating parts of the analysis

I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.

In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.