As system administrators we usually end up spending a good amount of time and thought on monitoring the systems we are responsible for. In this post I am not going to talk about how to monitor. Everyone talks about how to monitor, and what tools — Nagios, Munin, Monit, and so on. I think people tend to focus on that because it is often the more complex topic. However, I want to talk about what to monitor.

Honestly, I have never put much organized thought into what to monitor. I have always just tried to think of what is important and then make sure I monitor those things from both the outside and inside of my environment. This has served me fine in the past but this time around I want to do monitoring better. I have read a good amount of computer books and I am sure there are some that write about what to monitor. However, I want to start to look outside my direct field for inspiration. Most of the admins I have read about online monitor websites, email, and that sort of thing. But those are just websites and just email, and when trying to think of people who really need to get it right one thing popped right into my mind. The people who monitor nuclear power plants — I bet they feel the need to get monitoring right.

My research into this led me to the topic of reliability engineering and that led me to the topic of fault tree analysis. I combined those two and came up with the Fault Tree Handbook published by the office of Nuclear Regulatory Research of the U.S. Nuclear Regulatory Commission. It is a bit old, 1981 to be exact, but I figure these people take their monitoring seriously.

On Systems:

Before I get to what a fault tree is I am going to cover some of the highlights from the Fault Tree Handbook that I found relevant to monitoring and system administration. The first is the authors’ definition of a system:

A system is a deterministic entity comprising an interacting collection of discrete elements.

I think this definition has some interesting implications that the authors point out. The first is that we are interested in discrete elements. These can be parts, or systems themselves (subsystems). Identifying these parts and subsystems is our task when we decide what to monitor. Two primary things must be decided, what is the “external boundary” of the system and what will be the “limit of resolution”. For example with the external boundary, should I be monitoring seismic activity, or is that out of scope? The limit of resolution is simply how much detail do I need.  Do I need to monitor individual hard drives or just logical arrays.

The second interesting piece of this definition is that an “interacting collection” creates something that is more than the sum of its parts because the interactions must also be taken into account. These interactions can prove to be the most difficult to monitor, particularly in terms of performance, and is something I want to investigate more in the future.

Inductive and Deductive Approaches:

Inductive reasoning is reasoning from the specific to the general. Deductive logic goes the other way and is from the general to the specific. This is relevant when we try to think of what to monitor from a system failure perspective. If we take the deductive approach, we think of the most general case of failure, and then perhaps move onto the failure of a subsystem. From there, we try to think of what could cause that subsystem to fail. With inductive reasoning, we try to enumerate all the discrete items in the system and ask ourselves what happens if one or more pieces in the system fails. Both of these are valid and useful approaches, but fault trees are based in deductive reasoning. So on to what fault trees actually are.

Fault Trees:

A fault tree in its most basic form is a diagram that has a undesirable event at the root of the tree. Further down the tree is other event(s) that can lead up to that undesirable event. After reading the Fault Tree Handbook I went ahead and made my own “serverfault tree” with Visio (Larger Version):

The colors are meaningless and what actually has meaning is the shapes. A rectangle is an event that will be expanded upon. The circle is a basic event so it ends up being the limit of resolution of our fault tree. The diamond is an event that could be expanded upon but just isn’t for whatever reason. Lastly in this diagram there are those connector shapes. The inverted U shape is called an AND gate and the inverted V is an inclusive OR gate. What they mean is that all the items below them must either be OR’d or AND’d together to create the above event. So for example, if you look at the lowest leafs in my tree, you see there is an AND gate for the first and second hard drive basic events. This is because for RAID 5 two drives must fail for the array to fail. If we go one level higher in the tree there is an OR gate because with this particular server either the controller or the array (or both) must fail in order to create a failure of the disk subsystem.

Earlier I said that this tree is deductive because it is built from the top of the tree down. The tree is built from the most general case of a failure to the specific events that can cause it. However when building the tree I found an inductive approach helps as well. When I built mine I also thought of individual components and how they might effect the functioning of the server itself.

Why Bother?

In monitoring there are usually two general functions. One is to alert when something fails and record it. The other is to gather data about performance over time and record a history of that data which is particularly useful for capacity planning and trending. The fault tree is mostly useful when it comes to the component failure aspect. It highlights what must or ideally needs to be monitored. I think it also elegantly displays why these things must be monitored. The OR gates often imply that these things need to be monitored because they can cause failure. The AND gates show that monitoring this level can prevent failure because when one item is gone it should be fixed to reestablish redundancy.

When caught up in the details of configuring something as complex as Nagios it is understandable that the mind focuses. One might get so caught up in setting up the monitoring of a RAID array and configuring SNMP properly that the controller is completely overlooked. In the act of creating a fault tree we can pull ourselves away from the implementation details as system administrators which will allow us to brainstorm more creatively and effectively about what needs to be monitored. We can then go back and implement it.

My example is just a single server. If you run whole clusters you can build a separate tree, or extend one like my example, to cover the entire cluster.

If you noticed this form also shows where you do and don’t have redunancy. In general an AND gate will probably mean that there is some redundancy built into this particular part of the infrastructure. So you then have a picture of where you might need redundancy within the entire cluster if you build a fault tree or fault trees that represent the whole infrastructure.

I have set up complete network redundancy in the past using a combination of NIC teaming, multiple switches, HSRP, and BGP. When I tried to test that I got network redundancy right I took an inductive approach and simulated how redundancy worked when any of the particular pieces I could think of failed. This technique could be useful in developing test plans and analyzing redundancy that has been put into place. Also the implementation of this is relatively complex so again when an administrator is caught up in implementing it a flaw might be over looked in the bigger picture.

If you want to take this even further you can assign probabilities and possibly recovery time weights to each basic component to see where redundancy could best be implemented if you have a big enough organization to make this worth the time.

These could also be useful tools in troubleshooting just as monitoring itself is. When something has gone wrong and the pressure is on the fault tree can be used to see if what went wrong is an item in the fault tree. If it turns out the problem was not in the tree the item can be added to the tree for future implementation to help ensure that the mistake is not repeated. 

I see many benefits to the fault tree as a system administrator’s tool and I am adding it to my toolbox.

When the scope of the Stack Overflow engine was drastically increased, it became obvious that more hardware would be needed and that we would now have to put more emphasis on scaling.

Currently, everything lives at PEAK Internet in Corvallis, Oregon, which has worked out very well so far.  However, after much discussion, we came to the conclusion that we’ll be able to serve our globally distributed user base better if we at least move our heavier services to the East coast.  Since Fog Creek has had great experience with Peer 1, it seemed like a reasonable enough place to set up camp.  

We’ve decided to jump into two full-sized cabinets, each one with dual 40 amp power circuits pre-installed.  The current line of thinking is that we’ll spread our systems out evenly so that a serious disruption in one cabinet won’t take down service in its sister.  We want to be able to lose as many servers as possible and still be able to provide answers to you all.  How this actually works out in practice will be seen and I’m sure many revisions will follow as we learn.  

Each rack will contain two Rariton Dominion PX PDUs, one per power circuit.  These guys have a solid reputation and each unit gives us 20 receptacles for our servers: much better than the usual 8-10 receptacles that are often seen.  These units are manageable via SNMP, which allows us to monitor for spikes or other changes in activity over time.         

For basic data services, we’ll put two Dell PowerConnect 5448’s in each cabinet.  These are CLI-managed gigabit switches running a near-clone of Cisco’s IOS.  They’re great: you get basic trunking, VLAN, and SNMP support at a killer price.  Plus you can manage all of this by just ssh’ing into the switch itself.  Each switch will be interconnected with the rest (we’re protected from switching loops via STP, of course), and each server will have a connection to both switches within its parent rack.  

Now for the server hardware.  With only a few exceptions, our basic building block is the Dell R610.  These are amazingly flexible 1u boxes that are remarkably power efficient and capable of taking on almost any role assigned.   They also come loaded with the DRAC, a fantastic out-of-band management system that has saved me too many times to count.  We’ll be using these all over the place.

We’re going to run our own firewall / gateway layer on two Dell R610’s with an E5640 processor, 250GB of disk space at RAID 1, 8GB of RAM, and Intel NICs.  They’ll likely be running the OpenBSD operating system, which gives us PF for the firewall and CARP for high availability.     

There will be two load balancers built with the same hardware configuration as above, but running on top of FreeBSD.  It has been my experience that while OpenBSD is excellent for building firewalls, it hasn’t been road tested enough in other areas.  FreeBSD on the other hand, certainly has.  We’ll be using CARP again for high availability, and will run some combination of HAProxy and nginx to take care of the actual load balancing.

We’re expecting to deploy 10 web servers to start with (we currently run 6), all on the R610 platform and running Windows Server 2008 R2 Standard Edition.  They’ll have one Intel E5640, 250GB of disk space at RAID 1, and 16GB of RAM.

Two VMWare ESX servers will be built, the major difference being that these will have about a terabyte of usable storage at RAID 6 and 32GB of RAM.  

Two domain controllers will be in place, providing basic Active Directory services for all of this stuff.  They’ll be running on the 250GB of hard drive storage and 2 GB of RAM.

  

A syslog / monitoring server will run on a terabyte of storage at RAID 6 and 8GB of RAM.  All *nix syslog and Windows Eventlog data will be shipped over there for regular analysis and alerting if necessary.  It will be running FreeBSD.

A dedicated backup server will be in place to ensure that critical data is well protected.  We’re going to put this on an R610 with a 1.5TB of disk space and 16GB of RAM.  We’re mostly concerned about database backups, and this should give us considerable amounts of onsite storage.

The exceptions to the R610 rule are the database servers.  We’re going to put two in place, each on a Dell R710.  This shares the same guts as the R610, but is a 2u model and can house eight 2.5″ disks as compared to the six that the R610 can.  We’re going to load these up 64GB of RAM, a terabyte of storage at RAID 10, and are going to put two E5680 processors on board.

It should be understood that the above is not carved in stone: much of this will change during the actual deployment.  It’s great to lay out these big plans – they help light the way and keep you moving – but we’re not bound to any of it.  I tend to implement things organically, trying to build things in context of their environment.  What works, works. 

We’ll be posting more as things develop.

I began working with the Stack Overflow team at the beginning of May, and one of the first challenges I faced was to find a method to stop abusive users and bots from hammering the site.  While we’re happy to make way for legitimate search engines and users of our API, we need to ensure that the majority of our cpu cycles are going towards helping everyone find answers to their questions.  We needed a rate-limiting mechanism.  

After discovering that our tried-and-true load balancer did not have a native facility for rate limiting, I turned my attention towards nginx.  A quick scan of the documentation revealed that the limit_req_zone declaration should take care of things.  After a few tests with the development team, it was determined that it might actually work.  When any user surpassed the configured rate limit, a custom 503 error page would then be displayed.

Since HAProxy is already an important part of our stack, it didn’t make sense to rip it out.  Instead, I opted to slide nginx in front of haproxy, ultimately allowing us to add rate limiting to Stack Overflow with minimal effort and less risk to everyone.  If worse came to worse, nginx could easily be pulled back out of the stack without too much interruption.  I chose to break the work up into two phases: nginx without load balancing, and nginx with load balancing. 

I made the initial deployment on Sunday, May 16, and as you can see here, there were a few small glitches as our Monday morning traffic began to flow in.  Thankfully, Geoff’s quick thinking helped me see that we had exceeded the maximum amount of connections that the Linux kernel was configured to track, and I was able to adjust things appropriately.  After that, we had a few small bouts of slowness, which were overcome simply by increasing nginx’s worker process count and reloading the config.  I left this in place for a week, making extra sure that the new layer wasn’t going to have any adverse effects.

Rate limiting was then officially enabled on Sunday, May 23.  There was a bit of shakiness at first, as I did not enable the ‘nodelay’ feature.  This caused nginx to actually try and slow down offenders instead of just handing them 503s.  Once this was discovered, the feature was turned on and we were in business.

Over the past few days, I’ve been pouring over our http access logs, trying to determine if we have seen any amount of success.  I’m happy to say that we have: where I normally would see dozens of bots hitting us somewhere between 16,000 and 40,000 times an hour, I saw… none.  Victory for nginx! 

More to follow as we continue to tweak nginx and employ other systems to help us keep the channels clean…