The Three Perspectives
In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:
- Micro: “Ground Level”
- Meso: “Day to Day”
- Macro: “Seasonal”
These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.
From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.
I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.
Ground Level Monitoring
“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:
- Wireshark / TCPDump
- Web Logs (or other detailed logs)
- SQL Server Profiler
However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.
In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.
The Attributes of Micro Data
The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:
- Different sets of tools to process the data are needed
- Generally samples of high resolution data and not complete sets are used
- To correlate all of this, you will need to do some work because it will be different for every environment
Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.
One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).
Case Study: Web Logs and Perfmon Data
Data Analysis Platforms
Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.
Getting the Data
For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.
Exploring the Data with R
I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):
Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.
If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.
What is Next?
I don’t have a high resolution system deployed yet. So I need work on and think about:
- Scheduling Perfmon
- How to get the most representative samples
- Scheduling TCPDump and getting that data into R
- Scheduling Sar on the Linux boxes and getting that into R
- Automating parts of the analysis
I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.
In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.