Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.

How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.

Observability is the Foundation

Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:

  • Questions operators have about their systems can be quantifiably answered with minimal effort
  • Operators have rich mental models of how their systems function

When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.

Decision Making and Incident Preparedness

Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.

OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.

Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.

In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.

You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.

Other Benefits

Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.

Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.

Convinced? 5 Steps to Achieving Good Observability:

In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.

  1. Instrument your systems by publishing metrics and events
  2. Gather those metrics and events in a queryable data store(s)
  3. Make that data readily accessible
  4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
  5. Establish the resources to drill down into abnormal or out of bounds behavior

Each of these steps largely depends on the previous step to be successful.

1. Instrument your Systems

Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:

  1. Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
  2. Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.

Good Metrics also tend to have these properties:

  • High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
  • Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
  • Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.

It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.

2. Gather those metrics in a queryable data store(s)

This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:

  • Building dashboards
  • Enabling capacity planning
  • Allowing operators to explore the data and learn
  • Allowing people to invent cool stuff you didn’t anticipate

As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:

  • Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
  • Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views

3. Make that data Readily Accessible

If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:

  • A fast responsive UI to allow for operators to drill down and explore easily
  • Enables operators to create their own dashboards and graphs
  • Highlight problems

4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior

Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.

Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.

Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.

5. Establish the resources to drill down into abnormal or out of bounds behavior

The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:

  1. Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
  2. Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
  3. Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)

Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.

Use the science, Luke

If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.

The ServerFault Systems Administration team continues its growth with the addition of sysadmin icon Thomas Limoncelli.  You may know Tom from his books, Time Management for System Administrators and The Practice of System and Network Administration, or from his many conference appearances at events like LOPSA-East, the Cascadia IT Conference, and USENIX LISA.  Tom’s also been a ServerFault user, though references to his books outnumber the number of direct answers he’s supplied.

When we saw that Tom was just finishing up his time at Google as we were posting the next ServerFault opening, it was kismet.  Tom is the quintessential systems administrator and a great fit for our team.

I’ve been lucky enough to know Tom for the past couple of years and I am very excited to be able to work with and learn from him (and hopefully teach him a trick or two on our Windows stuff).  His The Practice of System and Network Administration was one of the first resources I had when I got started working as a systems administrator and really got my career headed in the right direction (well, if you consider getting to work with a small crack team of sysadmins and developers on one of the more dynamic environments out there “the right direction”).

Tom is joining Bart and George in working out of our (newly constructed) New York headquarters. Tom hails from New Jersey and frequently attends local LOPSA meetings.

Join me in welcoming Tom to our team!

Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.

eye-terminator-salvation2

We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

  • Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
  • Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
  • Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
  • Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

  • There are constraints of what is actually available to buy, and how much you can actually know what you are getting
  • Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.

drill_sergeant

When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

  • Gave us time to rectify some mistakes
  • Gave us confidence in our secondary datacenter since we failed over to it before the storm
  • Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.

ThingOfBeauty

Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

StackExchange welcomes our newest system administrator, Steven Murawski!

Steven Murawski Steven followed a bit of an eclectic path to systems administration, having started out operating a garden center, floral design and landscape business. After eight years he decided to move into law enforcement; he went to school for Police Science and worked at a local police department as a clerk, dispatcher and auxiliary officer. While he enjoyed shooting various firearms and driving at high rates of speed, he also began working with the police department’s IT systems, and slowly fell in love with systems administration. He became a member of LOPSA, Microsoft MVP, and continues to lead two local user groups (the Greater Milwaukee IT Pro User Community and Greater Milwaukee Script Club). Steven combined his love of systems administration with public speaking by presenting technology talks at various technology conferences and user groups.

Today Steven distracts himself from technology by playing with his eight month old son, puttering around with home-improvement projects and hurling pieces of lead through paper targets, although he is seriously considering once again picking up the mantle of host for a systems administration and Powershell podcast!

I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

  • Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
  • Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
  • Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
  • Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

Server Fault Survey!

George Beech

Like any good family, when we saw that our older sibling had something we wanted what did we do? Well, we whined and complained until we got it!

Stack Overflow recently finished their 2011 survey, and as soon as the results where announced, Server Fault being the proper younger sibling that it is wanted one too.

So I call all of you Server Fault users to come and take our survey, you wanted it and now you’ve got it! (oh and you really should get your sysadmin friends to take it too – the more data the better!)

Just like the Stack Overflow survey we will be releasing the data right when the survey ends!

At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

  • Monitoring allows you to send alerts if certain conditions are met,
  • It allows you to visualize trends in data,
  • Provides a method of assurance to the customer that their consumed services are guarded,
  • Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
A Large Network Operations Center

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.

 

One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.

 

Xzibit meme targeting monitoring systems

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

We have gone and done it, we’ve expanded our Systems Administration team to THREE people. I’d like to introduce you to Peter Grace who we have just brought on board as our third sysadmin.

This is the second company I’ve had the honor of working with Pete at, and I can say that he is an exemplary System Administrator. In addition to being a great sysadmin, Pete is a fellow gun enthusiast, which increases our zombie attack survival rate by 35%.

Pete and I have done many great things together in the past, and I’m very excited about what he brings to us and the great things he, Kyle and myself are going to do as we move forward doing our part to make the internet better.

Recently there was a question on the Programmers site asking “Why don’t all companies buy developers the best hardware?” This is actually a very interesting question and there was a good deal of discourse on the topic, as there always is when you get a group of professionals talking about the tools of their trade.

We here at Stack Exchange pride ourselves on not letting the technology get in the way of our devs. We want them to be able to do their job – which is writing code, and doing it well – with minimal hassle. Now, I have to take a brief moment to make a little bit of a note some of the things we do do not scale.

The philosophy that is maintained when getting a developer a new machine is a pretty simple one:

> If a dev is constantly struggling with their machine they aren’t getting work done, they aren’t happy, and they are producing bad code. All of these things are more expensive than a nice developer machine

As a sysadmin – and hence the guy in charge of getting the devs what they need you find out really quickly that they know what they want for the most part, and for those that you don’t ask if you give them a powerful machine with a lot of screen real estate they are generally very happy.

Since I mentioned earlier that there are some things we do here that aren’t all that scalable – in that they work for a company with 20 devs but not 100 i’ll split them up.

The Scalable

The base config that I get new devs in the NY office is:

  • Dell Optiplex 980 class tower
  • Max out the RAM
  • Best i7 Processor that I can get in it
  • SSD primary drive
  • 7.2k large (500GB – 1TB) Secondary Drive
  • 30″ Primary Monitor
  • 20″ Secondary monitor (turned sideways)

Every dev picks their own keyboard – we may give them the crappy one that comes with the machine to get them up and running, but they can request any keyboard/mouse combo they want (and … i do mean any).

The not so scalable

  • For our remote devs we get workstation replacement class laptops. Basically I go out and find the most powerful laptop I can get at the time they start, as well as a 30″ monitor to go with it. Not Scalable.
  • At least one dev in the NY office has a fully customized hand built machine with 2 30″ monitors. VERY not scalable.
  • Whatever they want – within reason

Basically, it boils down to get them what they need to get their job done. One of the biggest challenge to some people is that they do not have management’s buy-in to get the Devs what they need. I find that this is a sad state of affairs, but I have no real advice to offer – especially since I work somewhere that management has mandated great dev machines.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

  • When the test was run in UTC time and who ran it
  • Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
  • Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
  • Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.