Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.

How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.

Observability is the Foundation

Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:

  • Questions operators have about their systems can be quantifiably answered with minimal effort
  • Operators have rich mental models of how their systems function

When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.

Decision Making and Incident Preparedness

Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.

OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.

Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.

In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.

You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.

Other Benefits

Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.

Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.

Convinced? 5 Steps to Achieving Good Observability:

In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.

  1. Instrument your systems by publishing metrics and events
  2. Gather those metrics and events in a queryable data store(s)
  3. Make that data readily accessible
  4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
  5. Establish the resources to drill down into abnormal or out of bounds behavior

Each of these steps largely depends on the previous step to be successful.

1. Instrument your Systems

Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:

  1. Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
  2. Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.

Good Metrics also tend to have these properties:

  • High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
  • Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
  • Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.

It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.

2. Gather those metrics in a queryable data store(s)

This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:

  • Building dashboards
  • Enabling capacity planning
  • Allowing operators to explore the data and learn
  • Allowing people to invent cool stuff you didn’t anticipate

As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:

  • Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
  • Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views

3. Make that data Readily Accessible

If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:

  • A fast responsive UI to allow for operators to drill down and explore easily
  • Enables operators to create their own dashboards and graphs
  • Highlight problems

4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior

Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.

Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.

Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.

5. Establish the resources to drill down into abnormal or out of bounds behavior

The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:

  1. Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
  2. Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
  3. Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)

Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.

Use the science, Luke

If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.

If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

  • Monitoring allows you to send alerts if certain conditions are met,
  • It allows you to visualize trends in data,
  • Provides a method of assurance to the customer that their consumed services are guarded,
  • Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
A Large Network Operations Center

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.

 

One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.

 

Xzibit meme targeting monitoring systems

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

It has been my experience that many people simply buy a wireless access point, plop it down squarely next to their home cable/dsl modem, and assume that’s all they have to do to maximize their WiFi experience. Oh, were it to be so simple! I’d like to take a few minutes of your time today to cover some of the basics of what WiFi is, what it is and is not capable of, and how you as a SysAdmin or a home user can do a bit of detective work to help ensure your WiFi experience is less prone to issue.

RF Basics

Let’s take a moment and talk about Radio-Frequency Radiation. RF is a form of non-ionizing radiation where waves of energy radiate from a source and follow a predictable pattern based on the transmitter power and antenna. Radio waves are measured based on the size of the wave, and how frequently the wave oscillates. The frequency is measured in Hertz (Hz), or cycles per second.

Animated gif visualizing Hertz.  Image found on wikipedia, courtesy of Superborsuk.

Wavelength is the distance the radiation travels before it completes a single cycle. As we are mentioning travelling, we need to know the speed, right? This, my friends, is the speed of light.

C = f * λ , which translates to: Speed of Light = Frequency * Wavelength OR, if you're lazy, 300/Frequency in megahertz. Light travels approximately 300 million meters per second, we can drop a whole bunch of zeros from the equation and still be reasonably accurate.

WiFi signals operate on 2.4 gHz (2.4 billion cycles per second), and that means that one full wave travels around 12-13 centimeters before the waveform returns to its starting position relative to the axis in the graph. 802.11a and 802.11n operate on the 5gHz range, which would put the signal wavelength at 6 centimeters.

OK, but, why should I care about this when all I want to do is surf porn and play online games? The answer lies in the fact that if your antenna is not properly suited for these measurements, it won’t work that well. The antennas you get from your access point vendor are “suitable” but far from ideal.

Not many people realize it, but there is an aftermarket for antennas for access points. When people/companies buy commercial grade access points, they usually don’t include any antennas, as it’s assumed you’re going to get the proper antenna for your application.

So, what types of antennas are there and what are the differences?

Omnidirectional – These are the antennas that people are most familiar with. They will usually be oriented vertically, and radiate their signal on the horizontal plane in all 360 degrees. See below radiation chart which does a good job of visualizing how the energy travels out of a veritcal omni antenna.

Vertical omnidirectional antenna radiation pattern.  Image from wikipedia, courtesy of user LP.

Directional/Yagi – Directional antennas are designed to send a signal straight to a specific spot with pinpoint accuracy. If you’re trying to setup a WiFi link between your house and a neighbor down the street, you’d need a directional antenna. The Pringles Cantenna is an example of a homemade directional antenna. Commercial antennas more closely resemble old TV antennas that everyone seemed to have on their house back in the 20th century. The below radiation pattern does look a bit weird, but understand that the directional beam is designed to be highly selective of signals based on its relative orientation versus the target signal. This allows a directional antenna to receive and send to stations much further away than an omnidirectional antenna, which sends RF energy in all directions.

directional antenna radiation pattern.  Image is unattributed; if you own it contact me to remove or get credit.

patch – Patch antennas are normally flat antennas that are designed to radiate in a forward direction extremely well, with the signal attenuating sharply at the periphery. The radiation pattern below does have some similarities to the directional/yagi radiation pattern, but its lobe is more rounded in the forward direction. The patch antenna type is a good choice when you want to direct most of your energy in a particular direction but don’t necessarily want the pinpoint accuracy of a yagi.

patch antenna radiation pattern.  Image is unattributed; if you own it contact me to remove or get credit.

What blocks WiFi?

WiFi, operating in the 2.4ghz and 5ghz ranges, propogates in “line of sight.” Due to the short wavelength, the energy dissipates quicker if it is not channeled into a high-gain directional antenna. Consumer grade access points come with omnidirectional rubber duck antennas, which people usually orient vertically. If you look at the above radiation pattern, you’ll see that there is a void of energy directly above omnidirectional antennas when they are oriented horizontally.

All matter will attenuate RF energy to some extent as it passes along. The question on many people’s minds is what are the worst places you can install a wireless router or access point? Believe it or not, most materials in the home are not capable of attenuating your WiFi signal to a noticeable degree. In order for WiFi signals to be blocked effectively, they need to move through several layers of dense material in order to shed the energy required to become unusable. Some antenna manufacturers will quote how well the radio waves will propogate from a given antenna, as shown here for one of Cisco’s branded antennas:

The density of the materials used in a building’s construction determines the number of walls the signal must pass through and still maintain adequate coverage. Consider the following before choosing the location to install your antenna:
  • Paper and vinyl walls have very little affect on signal penetration.
  • Solid and pre-cast concrete walls limit signal penetration to one or two walls without degrading coverage.
  • Concrete and wood block walls limit signal penetration to three or four walls.
  • A signal can penetrate five or six walls constructed of drywall or wood.
  • A thick metal wall causes signals to reflect off, causing poor penetration.
  • A wire mesh spaced between 1 and 1 1/2 in. (2.5 and 3.8 cm) acts as a harmonic reflector that blocks a 2.4-Ghz radio signal. (NOTE: as a commenter below further explains, this type of mesh is common in plaster walls from the 1940s as well as in stucco applications.)

I once ran into an issue with a friend who had his wireless router installed in the basement, next to his cablemodem. He was having sporadic connectivity issues in a second floor room and asked me to come help diagnose. Sure enough, his room was directly above the wireless router, two residential floors below, and given that traverse and the location of his room in relation to the radiation pattern, there wasn’t enough RF energy propogating up into that location. The short-term answer for the problem was to orient his access point antennas horizontally, so that the radiation pattern is then set on its side, covering a wider swath of his house.

In a blog post to come, I will show you some methods you can use to help properly locate your access point and also help you decide which frequency your access point should operate on. Stay tuned, and as always your comments,criticisms and suggestions are always welcome!

In the never ending quest for better performance and response time, perhaps consistency is underrated. In web operations the desire is to make page loads go fast, because performance is a feature, or more simply — people like responsive web pages.

Probably the most common and simplest way to look at response time is to look at the average response time for the primary web request in a page load. Page loads also need to be looked at from the perspective of how they render in a browser (Which is how Google looks at it). This all makes sense, but as you push your load times faster and faster, the savings start to get smaller and smaller. For instance you start reducing the average time of certain requests from something like 1 second to 500 MS, then 500 to 100, then 100 to 80, 50, 30, 15, 10. Eventually these push to limits where you are less likely to feel the difference. It doesn’t mean it is time to stop, but I think in general it makes sense to shift your attention elsewhere for performance gains.

When this happens, a next logical step is often to go for a Content Delivery Network. The CDN can help reduce load times for people who are geographically far from the main servers. We did this back in May for our static content. Then what?

I think a next logical step is to start to shift attention to the consistency of your web server’s response time. By consistency I mean addressing those responses that take a lot longer than your average request time because of the server’s response time. In other words, fixing the outliers. To address these, is quite frankly, a pain. Why?

  • You need a decent sized sample of the response time of every request
  • At hundreds of requests a second, it is hard to pin down the cause of these outliers. It can be code, DB behavior, network blips, etc.

The best way I have come up with to measure this is to pull the response time from our web logs as our load balancer measures it from:

Server response time (HTTP mode only). It’s the time elapsed between the moment the TCP connection was established to the server and the moment the server sent its complete response headers. It purely shows its request processing time, without the network overhead due to the data transmission…

To be honest, I am not sure if this is the best way, but it seems like a reasonable start. I then filtered a sample of log requests so that I was only looking at authenticated users loading a question on stackoverflow.com that got a 200 response code from the web server (to ease in taking crawlers out of the mix). Then to express the outliers I used a box-and-whisker plot. If you have been wondering just what that image on the right is — this is what you are looking at. You can go read more about it, but in short, the dots are the outliers. When you look at this as a histogram these outliers will be the long tail (which in this case pushes the average to the right of the median).

I have been giving Marc Gravell some data and we were able to pin down a lot of these as collateral damage to a particularly demanding background thread that runs on each server. He is reworking this to help eliminate a lot of these.

I think keeping an eye on these outliers or the long tail of response is going to be tricky, but I also think it plays an important role in the quest for better performance.

The Three Perspectives

In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:

  • Micro: “Ground Level”
  • Meso: “Day to Day”
  • Macro: “Seasonal”

These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.

From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.

I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.

Ground Level Monitoring

“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:

  • Perfmon
  • Sar
  • Wireshark / TCPDump
  • Web Logs (or other detailed logs)
  • SQL Server Profiler

However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.

In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.

The Attributes of Micro Data

The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:

  • Different sets of tools to process the data are needed
  • Generally samples of high resolution data and not complete sets are used
  • To correlate all of this, you will need to do some work because it will be different for every environment

Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.

One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).

Case Study: Web Logs and Perfmon Data

Data Analysis Platforms

Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.

Getting the Data

For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.

Exploring the Data with R

I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):

Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.

The Difference?

If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.

What is Next?

I don’t have a high resolution system deployed yet. So I need work on and think about:

  • Scheduling Perfmon
  • How to get the most representative samples
  • Scheduling TCPDump and getting that data into R
  • Scheduling Sar on the Linux boxes and getting that into R
  • Automating parts of the analysis

I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.

In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.

Transfer rates and the number of packets you send are measured in units of a certain quantity of data per units of time. The unit of time that everyone is used to is the second. The standard quantity of data that is used in the networking field is bits and the standard time unit is seconds. So for example, the standard network interface these days is 1 Gigabit per second. So the quantity of data is a Gigabit, and the unit of time is a second. We call this the transfer rate. The key thing to remember is that this is a fixed ratio of data over time. Because of this, you can divide the ratio by any number you want to (Ignoring the complexities of the discrete properties of Ethernet frequencies, system clocking, etc). So, 500 Mbit over a half second is the same fixed ratio as 1 Gigabit per second.

The thing is though, in computing, a second is a really, really, really long time. This is important, because when we choose what unit of time to express this in, what we are doing is graph smoothing (It is sort of, although not really, like taking an average).

For example, we could transfer 900 Mbit in half of a second and another 100 Mbit for the other half of that second. How much data was transferred during that second? The answer is 1 Gbit. If we transfer 500 Mbit per half second and another 500 Mbit per the other half second — this is also 1 Gbit per second:. This effect is illustrated in these Megabits per half second graphs:

These two are clearly not the same thing, but when you express them as the amount of data transfered over a second they are. This is important because a 1 Gbit per second interface is also a 500 Mbit per half second interface — and a 500 Mbit per half second interface can’t transfer 900 Mbits per half second (I am ignoring any buffering effects, but in practice we have found this to be essentially true).

This effect is made even worse by most monitoring tools because most take samples every 5 minutes. So what you are really seeing is the transfer rate per 5 minutes converted to a per second rate. This sort of thing is why people say data can lie.

Why Should you Care?

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

We have bonded these interfaces using Intel Load Balancing (ALB/RLB) and for the most part our discards have gone away. We did this on all but one of our web servers for a while and found that the one that didn’t have the bonded interface had discards climbing while the others did not.

A second is a long time — be wary of trusting it too much to measure things.

Just about every introductory class for algorithms teaches you three primary ways to look at algorithm performance:

  • Worse Case Scenario
  • Average Case Scenario
  • Best Case Scenario (This is always followed up by why this isn’t very useful, but they mention it anyways).

You can use this same sort of analysis when thinking about your web performance. I have found that a histogram captures all 3 of these pretty nicely since it is a distribution. The X axis is the response time in milliseconds of a request, and the Y axis is the number of hits that had that response time . Our HAProxy web logs capture the response time of each request from the perspective of the load balancer. This is a nice perspective because it includes pretty much the full stack of a web request that is directly under our control.

We have started to insert our web logs into SQL server. Raw SQL is hard to beat for deep analysis, but I have also started to build a web front end which is particularly useful for generating graphs easily. With a web interface it is easy to filter on certain criteria and you can easily see the distribution for a particular page or a certain client like Googlebot:

(The X axis is the server response in MS, and the Y axis is number of hits that fell into that response bracket)

A crawler will also be a sort of worst case response time perspective on another level because because crawlers cause more cache misses. Clients will generally perform much differently. For example looking at response time for user agents with “Chrome” in them the response time has a very different shape:

In general, I always find it is best to think about how you are viewing your data, and if it is the best way to summarize what you really care about. Average is useful, but it just isn’t the complete picture.

We recently changed the NICs in our web tier and primary database servers from Broadcom to Intel based NICs based on some … issues we had been having. After we put them in they worked reasonably well, but we knew that they could be faster and push more data. When I started to dig into just what we could do to tweak the pleathora of settings for the new NICs I found a few settings that would probably help a little bit, and one technology that could help us out tremendously.

Changes at a glance

  • Turned on Intel I/OAT
  • Adjusted Send and Recieve buffers to 2048 (max allowed)
  • Turned off interrupt moderation
  • Increased the Receive Side Scaling Queues from 1 to 4

The long version

The first and biggest change is that we turned on Intel’s I/OAT technology. It’s a collection of different techniques that work together to improve the performance of your host networking, as defined on Intel’s website:

  • Intel® QuickData Technology — enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput.
  • Direct Cache Access (DCA) — allows a capable I/O device, such as a network controller, to place data directly into CPU cache, reducing cache misses and improving application response times.
  • Extended Message Signaled Interrupts (MSI-X) – distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance.
  • Receive Side Coalescing (RSC) — aggregates packets from the same TCP/IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing.
  • Low Latency Interrupts — tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency.

The catch is you need to be running a full Intel hardware stack. Your CPU, Motherboard, BIOS, NIC and OS all need to be compatible with the technology to be able to use it. Once you have the right stack in place, you might have to turn on a BIOS option, but that’s it no tweaking or poking to make it just right, it’s just right out of the box. Turning it on was as simple as flipping a BIOS setting, aptly named “Intel I/OAT.”

Performance tuning options

The performance options are all exposed via the PROSet utility making it nice and easy to change, no need to go digging into the registry for some esoteric key that may or may not be there. For each of them there is a trade off you need to consider. Some of the options will increase host CPU, some will cause higher host memory usage. To find the right value for your systems you really need to evaluate your overall situation and see if the trade offs are worth it.

Send and Receive Buffers

The Send and Receive buffers where set to the maximum allowed value of 2048. The trade off here is that you will consume more host memory. For us this is not a big deal since we have a lot of RAM on our boxes. Also, we had been seeing a good deal of Zero-window TCP packets when investigating our network so we needed to increase the buffer anyway.

Interrupt Moderation

The Interrupt Moderation feature was disabled. This feature allows you to have the NIC throttle the number of interrupts to the CPU which will limit the number of CPU cycles used by the NIC interrupts. Turning this off will increase your CPU usage, but it will also prevent packets from sitting there waiting for an interrupt to be proccessed. The increased cpu is a pain point for us right now (we are working on fixing that) but I believe it’s worth it.

Receive Side Scaling Queues

Receive Side Scaling is a technology that allows you to process a TCP connection across multiple cores. This allows for more efficient cache and processor usage when you TCP connection is not tied to a single core. When you are using this feature it will only use real physical cores to process TCP connections, you are not able to use this with hyper thread cores.

Receive Side Scaling Queues are essentially buffer space that is used between the NIC and the CPU when you are using Receive Side Scaling. This is another setting that has a trade off between host CPU and performance. I opted for the trade off, and increased the queues from 1 to 4 queues.

Additional notes

  • Since I/OAT needs BIOS support and the BIOS on our web tier was woefully out of date anyway I updated to the latest BIOS for those machines
  • I updated the Intel PROSet utility to verion 16.1 from 16.0

We are less than twenty four hours into using these new setting, but everything looks much much better through our peak traffic today. So far we are very happy with the results of these changes.

A Network Administrator’s View:

When digging into some packet dumps to try to solve some issues I was seeing with our Broadcom network cards, something else caught my eye. When looking in Wireshark to see if there were TCP retransmits I didn’t see any in my capture but I did see a very large amount of TCP zero window messages between our web tier and our Database tier.

To follow this you need to be familiar with TCP flow control, so I will briefly cover this. Since TCP is full duplex, each side of the connection is both a sender and a receiver. However, you will often have one side doing more of one then the other. In this case with our sites it is the SQL server backend that plays the role of the sender and the web tier is the receiver. The reason for this is that the web tier just sends a database query which will be short, and the database server will send back the results of that query which will generally be larger than the query itself.

The rate of data transfer is controlled by the receiver telling the sender how much data it can receive. The amount of data that can be received is called the TCP Window. This window shrinks as the network buffers fill up. If the window fills up faster than the application retrieves the data from the network buffers then eventually the receiver will let the sender know that is can’t receive any more data for the time being. TCP informs the sender that it can’t receive more data by sending a TCP packet where the window size is zero — this is our zero window message. What this means in our case is that the sender (SQL server) is sending data to the receiver (the web servers) faster than they can process it.

So as a network administrator, if I don’t want to just blame the application, I look to what I can fix on the network side. One cause of this would be that if there is a lot a latency between the web tier and the database tier than the window might be too small. To check this the simplest way was to send pings up to the size of the MTU with the don’t fragment bit set and make them as rapid as possible. I did this but only saw peaks of 1-2 MS latency. Even if we take a view that the performance is worse than measured, the bandwidth delay product for this latency is ( (RWIN in Bytes)/(Latency) * 8 = (Max Throughput in Bits) ):

(65535/0.003) * 8 = 174,760,000

So this didn’t really seem to be the issue here since the bandwidth is lower than 174 mbit/s. Also, in Windows Server 2008 R2 there isn’t much you can do to enlarge the default window by using window scaling because Windows automatically controls this.

The other theory I had is that maybe somehow the network stack or network driver is not letting the application know that there is data to be retrieved fast enough. CPU usage is moderate so I figured that it was not a lack of processing power the web servers. The way the network stack will inform the application that there is data in the buffers is by sending an interrupt. Because at gigabit speeds interrupts can start to take up a lot of CPU power there are several tuning options for this. One option is to dedicate these interrupts to a certain core or group of cores. Another option that the NICs have to keep the interrupt CPU load low is interrupt moderation, this dampens the rate of interrupts by batching them. I tried tunning these various options to make the interrupts more frequent but I still saw a high rate of zero window messages.

My skills as a network administrator pretty much hit a wall at this point and I didn’t get any network level answers on Server Fault that solved this issue for me. Next I turned to Stack Overflow to see if there was maybe a way to have .NET tell Windows to increase the size of the TCP window. My theory was that if the TCP window was bigger than it might stop bottoming out as it does in this graph of the average window size over time during my capture:

A Developer’s View:

When I asked about Speeding up the rate that IIS/.NET/LINQ retrieves data from the network buffers on Stack Overflow the pieces started to fall together with Remus’ answer. I wasn’t sure if what he was saying was the case, but I now had something to run with to try to get more information. With this information, I put on my admittedly somewhat shabby DBA hat.

A DBA’s View:

To verify that this might be the case I used the queries I had learned from Professional SQL Server 2008 Internals and Troubleshooting to view the SQL DMV of top wait times. One of the top wait times was async_network_io. This SQL server wait type means that SQL server has to wait because the client is not ready to receive all of the data it is sending. The problem with this DMV view is that it only shows total times since SQL server was restarted, and I needed to see which particular queries were causing the waits. So I turned to dba.stackexchange.com to try to find out how I could find the queries causing network waits safely in a production environment. The answers provided me with queries that I could run to take snapshots of queries. There were queries that frequently showed up with async_network_io wait times. I saw one query over and over again with 200-800MS of network wait. The query with “SELECT TOP 3000″ and a whole wall of fields after it raised my eyebrows as that sounded like a lot of data to be sending back to the web server.

Not being that much of a DBA (at least, yet) this was about the end of the road for me. So I sent the top offenders I found to the developers and Brent Ozar.

Back to the Developer’s View:

Remus’ original answer had two theories for what might be going on:

  • The client (web tier) was requesting more data than it should
  • There was waiting going on while processing the data before fetching it all

The Top 3000 query was clearly more data than was probably needed and was a query constructed by LINQ. A large query in some ways make sense at first because the results of this query were aggressively cached. Also, for a while now our web tier has had CPU power to spare so moving processing to the web tier appears to be a good thing to do. However, returning large data sets to the web tier usually won’t work well, at least as part of a user request, due to the high network penalty.

The second theory is that a DataReader is being used to read the data one record at at time, and something is performed on each row before fetching the next record causing wait time between each query. I am not aware of any instances of this for our large queries yet. If there are such queries the solution might be to use a DataSet which would fetch all off the rows before processing them.

So the solution was to move the query to a background thread so it won’t slow down user response time, and of course make the query more limited in the amount of data it returns.

Back to the Network View, Meta is Murder:

The most shocking thing is that after this query was adjusted the amount of data being sent from the database sever dropped about 20mbit/s (Notice the difference between Tuesday and Wednesday during peak hours):

So was this query really pulling this much data even though it isn’t running that frequently? The answer is both yes and no.

Since TCP/IP and Ethernet carries overhead for the headers part of the data going over the wire is just meta data added by the network. The minimal amount of TCP/IP and ethernet overhead is (See this page for more information):

(1500-40)/(38+1500) = 94.9285 % IPv4, minimal headers

So at an optimal window TCP window size each packet will have the maximum amount of user data of 1460 bytes (without jumbo frames/vlan tagging/etc). The 78 bytes of overhead in this case is about 5% of overhead (78.00/1538.00). When our network is not hitting zero window messages the window sizes were often around 200 bytes. I made a histogram of my capture to show just how often it was in the range of small windows (resolution isn’t there to show it, but most of it is around 200):

The window size of a TCP packet will be the size of payload data minus 8 bytes since the window size is everything beyond the acknowledgement number in a TCP packet (I might be off on this calculation, I could not find a direct reference to verify this). So with a window size of 200 bytes we are sending 194 bytes of user data in a packet. So with this we have 70 bytes of overhead and 194 bytes of user data which is about 27% of overhead.

So when transferring about 100 mbit/s of user data you would only see about 5 mbit/s of extra data in the SNMP octets counters with a good window size, so from the SNMP view the transfer rate would be 105 mbit/s. When transferring 100 mbit/s of user data with a window size around 200 bytes there is 27% overhead and you end up with about 27 megabits a second of overhead for 127 mbit/s of traffic from the SNMP view. This ignores any overhead provided by the application level protocol that the web tier uses to speak to SQL server.

This pattern of window sizes is referred to as “Silly Window Syndrome” since the meta data can start to overtake the actual amount of user data being transfered. This overhead explains the large drop in database traffic beyond just reducing the amount of query data returned.

Lessons Learned

I think the biggest lesson is that the full view of many problems is missed unless each person in the team has at least some understanding of what is going on from other team members views. Also, the communication between different specialists is needed to solve many issues. In this case what looked initially to me like a networking problem was actually a symptom. Trying to attack the problem solely as a network administrator was treating the symptom, not the disease.

From a technical standpoint the difference between having the web tier and SQL servers on the same box compared to having a network connection between them is important. Things may work well on a single server, but when the data needs to be moved over the network shifting the load to web servers might not always work.

We still have a good amount of zero window packets going back and forth, so although the worse offenders have been mitigated I believe there is still might be work to do.