As system administrators we usually end up spending a good amount of time and thought on monitoring the systems we are responsible for. In this post I am not going to talk about how to monitor. Everyone talks about how to monitor, and what tools — Nagios, Munin, Monit, and so on. I think people tend to focus on that because it is often the more complex topic. However, I want to talk about what to monitor.
Honestly, I have never put much organized thought into what to monitor. I have always just tried to think of what is important and then make sure I monitor those things from both the outside and inside of my environment. This has served me fine in the past but this time around I want to do monitoring better. I have read a good amount of computer books and I am sure there are some that write about what to monitor. However, I want to start to look outside my direct field for inspiration. Most of the admins I have read about online monitor websites, email, and that sort of thing. But those are just websites and just email, and when trying to think of people who really need to get it right one thing popped right into my mind. The people who monitor nuclear power plants — I bet they feel the need to get monitoring right.
My research into this led me to the topic of reliability engineering and that led me to the topic of fault tree analysis. I combined those two and came up with the Fault Tree Handbook published by the office of Nuclear Regulatory Research of the U.S. Nuclear Regulatory Commission. It is a bit old, 1981 to be exact, but I figure these people take their monitoring seriously.
Before I get to what a fault tree is I am going to cover some of the highlights from the Fault Tree Handbook that I found relevant to monitoring and system administration. The first is the authors’ definition of a system:
A system is a deterministic entity comprising an interacting collection of discrete elements.
I think this definition has some interesting implications that the authors point out. The first is that we are interested in discrete elements. These can be parts, or systems themselves (subsystems). Identifying these parts and subsystems is our task when we decide what to monitor. Two primary things must be decided, what is the “external boundary” of the system and what will be the “limit of resolution”. For example with the external boundary, should I be monitoring seismic activity, or is that out of scope? The limit of resolution is simply how much detail do I need. Do I need to monitor individual hard drives or just logical arrays.
The second interesting piece of this definition is that an “interacting collection” creates something that is more than the sum of its parts because the interactions must also be taken into account. These interactions can prove to be the most difficult to monitor, particularly in terms of performance, and is something I want to investigate more in the future.
Inductive and Deductive Approaches:
Inductive reasoning is reasoning from the specific to the general. Deductive logic goes the other way and is from the general to the specific. This is relevant when we try to think of what to monitor from a system failure perspective. If we take the deductive approach, we think of the most general case of failure, and then perhaps move onto the failure of a subsystem. From there, we try to think of what could cause that subsystem to fail. With inductive reasoning, we try to enumerate all the discrete items in the system and ask ourselves what happens if one or more pieces in the system fails. Both of these are valid and useful approaches, but fault trees are based in deductive reasoning. So on to what fault trees actually are.
A fault tree in its most basic form is a diagram that has a undesirable event at the root of the tree. Further down the tree is other event(s) that can lead up to that undesirable event. After reading the Fault Tree Handbook I went ahead and made my own “serverfault tree” with Visio (Larger Version):
The colors are meaningless and what actually has meaning is the shapes. A rectangle is an event that will be expanded upon. The circle is a basic event so it ends up being the limit of resolution of our fault tree. The diamond is an event that could be expanded upon but just isn’t for whatever reason. Lastly in this diagram there are those connector shapes. The inverted U shape is called an AND gate and the inverted V is an inclusive OR gate. What they mean is that all the items below them must either be OR’d or AND’d together to create the above event. So for example, if you look at the lowest leafs in my tree, you see there is an AND gate for the first and second hard drive basic events. This is because for RAID 5 two drives must fail for the array to fail. If we go one level higher in the tree there is an OR gate because with this particular server either the controller or the array (or both) must fail in order to create a failure of the disk subsystem.
Earlier I said that this tree is deductive because it is built from the top of the tree down. The tree is built from the most general case of a failure to the specific events that can cause it. However when building the tree I found an inductive approach helps as well. When I built mine I also thought of individual components and how they might effect the functioning of the server itself.
In monitoring there are usually two general functions. One is to alert when something fails and record it. The other is to gather data about performance over time and record a history of that data which is particularly useful for capacity planning and trending. The fault tree is mostly useful when it comes to the component failure aspect. It highlights what must or ideally needs to be monitored. I think it also elegantly displays why these things must be monitored. The OR gates often imply that these things need to be monitored because they can cause failure. The AND gates show that monitoring this level can prevent failure because when one item is gone it should be fixed to reestablish redundancy.
When caught up in the details of configuring something as complex as Nagios it is understandable that the mind focuses. One might get so caught up in setting up the monitoring of a RAID array and configuring SNMP properly that the controller is completely overlooked. In the act of creating a fault tree we can pull ourselves away from the implementation details as system administrators which will allow us to brainstorm more creatively and effectively about what needs to be monitored. We can then go back and implement it.
My example is just a single server. If you run whole clusters you can build a separate tree, or extend one like my example, to cover the entire cluster.
If you noticed this form also shows where you do and don’t have redunancy. In general an AND gate will probably mean that there is some redundancy built into this particular part of the infrastructure. So you then have a picture of where you might need redundancy within the entire cluster if you build a fault tree or fault trees that represent the whole infrastructure.
I have set up complete network redundancy in the past using a combination of NIC teaming, multiple switches, HSRP, and BGP. When I tried to test that I got network redundancy right I took an inductive approach and simulated how redundancy worked when any of the particular pieces I could think of failed. This technique could be useful in developing test plans and analyzing redundancy that has been put into place. Also the implementation of this is relatively complex so again when an administrator is caught up in implementing it a flaw might be over looked in the bigger picture.
If you want to take this even further you can assign probabilities and possibly recovery time weights to each basic component to see where redundancy could best be implemented if you have a big enough organization to make this worth the time.
These could also be useful tools in troubleshooting just as monitoring itself is. When something has gone wrong and the pressure is on the fault tree can be used to see if what went wrong is an item in the fault tree. If it turns out the problem was not in the tree the item can be added to the tree for future implementation to help ensure that the mistake is not repeated.
I see many benefits to the fault tree as a system administrator’s tool and I am adding it to my toolbox.