The Three Perspectives

In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:

  • Micro: “Ground Level”
  • Meso: “Day to Day”
  • Macro: “Seasonal”

These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.

From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.

I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.

Ground Level Monitoring

“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:

  • Perfmon
  • Sar
  • Wireshark / TCPDump
  • Web Logs (or other detailed logs)
  • SQL Server Profiler

However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.

In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.

The Attributes of Micro Data

The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:

  • Different sets of tools to process the data are needed
  • Generally samples of high resolution data and not complete sets are used
  • To correlate all of this, you will need to do some work because it will be different for every environment

Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.

One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).

Case Study: Web Logs and Perfmon Data

Data Analysis Platforms

Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.

Getting the Data

For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.

Exploring the Data with R

I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):

Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.

The Difference?

If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.

What is Next?

I don’t have a high resolution system deployed yet. So I need work on and think about:

  • Scheduling Perfmon
  • How to get the most representative samples
  • Scheduling TCPDump and getting that data into R
  • Scheduling Sar on the Linux boxes and getting that into R
  • Automating parts of the analysis

I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.

In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.

  • Joe P

    This is great. I wonder, is there any chance you could show us some of your R code (or are you strictly using the GUI?). I’d love to see how you’re using R with this data.

    • Joe P

      I just noticed that you actually linked you R script. Do you primarily write it by hand or generate it?

      • You can auto generate R scripts? 😛

        I am just getting started out in R. So that script is really me just posting the interactive stuff I run in R-Studio into a script so it is easy to rerun on new data.

        Proper R scripting will probably take me a little while. Thankfully we run a site which is pretty handy for my R questions when I get stuck 😉

  • Jim

    Have you looked at stuff like graphite ( particularly the way Etsy uses it: ? It’s nice cause it’s so unintrusive – you just pump your data to a listener (netcat or via common library) and it automatically saves your data in an rrd style DB. There’s a robust graphing front end in which you can compare various metrics on the fly. R is definitely one of the best visualization tools out there but for pure ease of use, I figured graphite/carbon/statsd/logster was worth mentioning.

    • Hi Jim,

      We have looked at it a couple of times. I think that fits more into Day to Day monitoring I will talk about in the next post for this series. 

      It looks really interesting, but because it doesn’t really have a lot in the way of templates, views, logins, etc — we just don’t really have the time to implement it in its current state IMO. 

      Maybe in a year or so.


  • Mr Wolf

    Is there a reason you chose R?  It seems like a language without broad applications.

    I’ve been doing some statistics work for one of the places I work, and using Python for it. Python has Numpy/Scipy, which are libraries for matrix math, statistics, and scientific computing, and MatPlotLib, which does plotting/graphing.

    The advantage is here that I can embed the data visualizations in python programs (I use Wx, matplotlib supports Wx/Qt/Tk), and then update them in realtime, if I need it. Also, it makes for nicer end-products.

    The best thing, by far, is that it’s a language I already know. Dealing with statistics is enough challenge ,without having to do it in a language I have never used.

  • Jim

    By the way if you’re exploring R, I highly highly recommend Nathan Yau’s book – Visualize This:

    Covers the basics on statistics in general as well and showed me some really novel ways to look at data sets. I’m kinda siding with Mr. Wolf said however; I’m still not sure that R is conducive to really large sets like weblogs tend to be. There are other tools out there that can lay out graphs side by side which allow you to do simple visual correlations. We use Zenoss for all of our stats – windows and linux – and its custom reports allow us to discover anomalies in graphs rather quickly.

  • Doug Y’barbo

    great post! i have a feeling that if i can persuade our NetOps guys to use R instead of Splunk, they will get more insight, so your use cases were very interesting. (as an aside, i might suggest, if you are not doing it already, that you want to decompose your time series data (which R is very good at) to remove the trend, cyclical and regime shift components; as you might imagine, these effects are usually so large that they can completely conceal even significant system modifications (e.g., adding another server).