The League of Professional System Administrators, LOPSA, is a non-profit organization aimed at advancing the profession of system administration. Like Server Fault, they want to see the knowledge and skill of system administrators expand. To this end, they do several important things which include local meetups and regional conferences. Also, both Server Fault and LOPSA are vendor neutral, we don’t care if you are a Cisco Network administrator or Linux administrator, we both want to see the field of system administration united.

So, why is this important to you? Server Fault has partnered with LOPSA to give 40 of our top users free memberships to LOPSA.

Both George and I have started to attend our local monthly meetings in New York (LOPSA-NYC) and Boston (Back Bay LISA). I asked George about how he felt about his LOPSA-NYC Meetings:

LOPSA-NYC, even though it is in it’s infancy, has been a great place for learning about methods and technologies I don’t get to interact with on a daily basis. That alone is a great thing, since – in my opinion – the goal of a sysadmin should be to learn about more than just the narrow set of technologies presented to them in their day job. A second, and some including me might argue a more important benefit out of going to the lopsa-nyc meetings is that I get to meet and talk to real people (err sysadmins, but close enough). The time after the talks is really where the advantage of LOPSA shines through, you get to talk to people that have seen the same issues as you, people who may have a new insight in a problem that you have been having that shines the light onto the solution.

LOPSA also puts on local conferences, out in Seattle they put on The Cascadia IT Conference. They also recently had PICC in New Jersey where both George and I gave a talk about Stack Exchange’s architecture.

Whether you call it DevOps, system administration, or system engineering our field is growing as a profession every day. Through the knowledge shared at conferences and local meetings like Cascadia and PICC, LOPSA’s mentor program, and Server Fault we constantly advancing, and these memberships are meant to be a part of that. We hope you are interested, and can take advantage of local chapters or decide to start your own. See this question to put your name in for a membership — at the end of a week or two we will sort by reputation earned this calendar year, take the top 40, and pass your email address on to LOPSA for your membership.

Look at your infrastructure and the way you handle your operations and ask yourself:

“Are we more like the Borg or the Federation?”

If you are not sure, one way you can tell is to look at the pieces of your infrastructure and see if they have personality. If a server has personality, then it is more like the Enterprise. There may be sister ships, like the USS Yamato, but it has its own personality because Geordi has made improvements and its crew has made it unique. Disgusting.

Why disgusting? Personality is the antithesis of scalability. With a large number of servers, each with their own personality, you will need more administrators, and a chaotic management structure. The Borg are the good guys. This is because of some of the distinct advantages of being more like the Borg:

  • In the Borg collective, individual drones and ships can be destroyed without much thought. They are easily replaced.
  • There are different types of Borg vessels, cubes, spheres, tactical cubes, but within these types, there is no personality.
  • Improvements that are assimilated become part of their whole collective making them highly adaptable

In IT operations, your servers should be drones that are easily replaced. You may have different classes of servers, but if they are all the same within each class they are easier to manage. For example, if you have a different brands of hard drives and RAID cards in each server, you will have spend time figuring out the settings, tweaks and compatibility issues for each of these different types. You will also have to track updates for each of these two different things and figure out methods of deployment for each of these types if a vendor doesn’t provide tools to manage these. With configuration and centralized management you can update and adapt improvements to all your servers rapidly. Deploying a change to all servers is just a script or a policy, not a manual process for each server. If you hand code each configuration file and manually deploy software they will start to become different over time when mistakes are made — this is one of the main ways servers develop personality.

At Stack Exchange, we have done a good job at achieving this with our web tier. We do have 1 staging web server, but for our other 9 web servers, the seventh is just seven of nine. George has made a deployment process which includes everything we need, making any web server disposible, and a new one ready for assimilation. Should one of them be destroyed, the others will automatically take over without concern. However, in some areas we are still more like the Federation. We have 4 database servers, and we have noticed areas where our db01/db02 pair is unlike our db03/db04 pair. If one of our primary servers fail, we will mourn the loss as we carry out a manual fail over process. If we were more like the Borg in this area, we could initiate self destruct on one of these servers without a care as the Borg queen would do.

The Borg are the role model, not the Federation. Next time you look at a server that has personality, your first thought should be: “You will be assimilated.”

At a recent talk at PICC, part of my introduction was:

“Hello and thank you for coming to our talk. This is George Beech and I am Kyle Brandt, and we are the two sysadmins at Stack Exchange…”

While “two” is the technically correct answer going by our self-anointed titles, it isn’t really the whole picture.

The Sysadmin Continuum

In reality here at Stack Exchange we all wear a lot of hats. While George and I may own the system administration tasks, we work together with our developers who are pretty accomplished sysadmins in their own right. If it were just George and I soely doing the work that could be considered system administrator work, we would probably have a pretty tough time of it at only two people.

In reality we are continuum. System administration is just a spot on the continuum where George and I spend most of our time. However, a lot of our developers spend a lot of time there and travel the same road.

Some recent examples:

  • Jarrod Dixon worked with me on developing a log daemon for inserting HAProxy syslog data into SQL Server
  • Nick Craver frequently hops on Dell and makes hardware recommendations
  • Jeff Atwood helps us with SSD research
  • Sam Saffron works on SQL restore scripts
  • Geoff Dalgas racks servers out in OR and troubleshoots our NAS
  • Ben Dumke and Jeff Atwood do some CDN research
  • Jeff Szczepanski, our Vice President of Products, analyzes packet dumps

Because of the spirit of shared responsibility, we get to take advantage of everyone’s knowledge and experience and we are all better for it.

So in the end, “How Many System Administrators does Stack Exchange Have?” is a question I can’t really answer.

A couple of weeks ago we had one of our edge routers go down on us. Nothing bad happened, failover to our secondary router work just as expected. Now, we saw something wierd this week when we looked at the internal interface graph for our secondary router.

I’ll give you a second to try and see what we saw – although I don’t think you’ll need a whole second to see something very very strange going on with this router. That’s right, there is a whole lot of outbound traffic on this router, but ZERO inbound traffic. The next question we had was what could possibly be causing this? I really don’t think that I could be anything good.

You can clearly see on the graph where we failed back to our primary router. But, after that there is still a ton of traffic that is traversing our secondary router when there should be very little traffic going through there.

After some digging around we found that our Windows servers had the wrong ARP address for the VIP of our routers. That’s right, Windows still had the wrong ARP address after the fail back to the primary router, it even had the wrong ARP address days later.

How could this be possible? I was stumped after some digging around it seems that Microsoft changed the way that the network stack handles Gratuitous ARP packets (GARP packets) with Windows Vista/2008 RTM. This change has persisted through to Windows 7 and Windows 2008 R2.

What is Gratuitous ARP?

Gratuitous ARP is when a system sends out an ARP packets announcing to all system what it’s MAC address is. You generally see these in HA environments that make use of Virtual IPs that can move back and forth between machines. You will normally see a machine issue a GARP packet when a fail over event occurs and the new machine picks up the VIP. Wikipedia ARP article for more info on the ARP protocol

What did Microsoft Change?

There is actually very little information out there about windows and how it handles GARP. The best resource i’ve found that gives a very good overview of the new windows networking stack is a very well written technet blog. About 3/4 of the way down there is a section named “Changes to ARP cache updating” within this section lies the answer to all the mystery of our weird network bandwidth.

>First, a Windows Vista or Windows Server 2008 will not update the Neighbor cache if an ARP broadcast is received unless it is part of a broadcast ARP request for the receiver. What this means is that when a gratuitous ARP is sent on a network with Windows Vista and Widows Server 2008, these systems will not update their cache with incorrect information if there is an IP address conflict.

>Additionally, when a gratuitous ARP is sent by a Windows Vista or Windows Server 2008, the following change has been made – the SPA field in the initial request is set to This way the ARP or neighbor caches of systems receiving this request are not updated. So, if there is a duplicate IP address, the receivers do not need to have their cache corrected.

The question is why is this such a big problem for Microsoft? Well the answer is they have hi-jacked the GARP packets for their Address Conflict Detection mechanism. You know that pop-up that says “Another machine on this network has been detected with the same IP address”. With previous version of windows they had the same mechanism, but still respected the normal GARP packets, thus there would sometimes be an issue with Windows systems updating their ARP cache with invalid data. They fixed it by breaking GARP.

Why is this a problem?

Beyond the issue we have seen with Windows 2008 not respecting GARP packets this can cause other wierd problems. One example I can think of off the top of my head, is that for HA systems that use GARP to facilitate moving the VIP when a system goes down is that you will now have to wait for the OS to timeout the neighbor cache. This will add more time to your fail over, possibly causing things that are expecting a quicker fail over to break.

Is there a Fix?

I have not been able to find a fix for this. Although there is very little information out there on Windows networking at that low of a level. If you know of a fix to this issue I’ve started a question on Server Fault.

In the life of a company and in the life of a person there is a line that gets crossed when it comes to being organized. When you start out in a business basic documentation and good organizational systems are often optional.

To be clear, I am not saying that they should be optional, rather, that from my experience this is just the reality of the situation. There is usually only a couple of people working on something, so they can always ask each other things and the amount information is small enough that people can mostly keep it in their heads. Also, to be more clear, I don’t really think this is the best way to do things, by its very nature it is inefficient. But in a fast moving startup, it is easy to give into the temptation to skip some of the niceties and get to the next project.

But then one day, you start to get close to the Organizational Event Horizon or OEH. The way you will realize it is by the way it feels, it is the feeling of being Spaghettified. In other words, you get stretched beyond what you can handle. Before you cross the OEH, good organization is about being more efficient and getting more things done. But once you have crossed the OEH, organization becomes the difference between success and failure. Important and critical tasks will start to get missed, and the stress of all the different things going on will tear you apart. If you don’t start to deal with the situation before crossing the OEH, failure in inevitable.

How To Handle It?

I feel like at least on the sysadmin things here at Stack Exchange I am starting to feel the pull. To fix this I think the key is to implement a few basic systems for organization that are based on two related principles: The KISS (Keep it simple stupid) and the “stub” principle. The idea behind both of these is to have the minimal amount of resistance to actually start using documentation and organizational systems. The KISS principle is well known. What I call the “stub principle” is something I picked up from wikipedia and Clay Shirky’s Here Comes Everybody:

“…as long as the experts did nothing (which, on Nupedia, is mostly what they did), nothing happened. In an expert-driven system, an article on asphalt that read “Asphalt is a material used for road coverings” would never appear, even as a stub. So short! So uninformative! Why, anyone could have written that! Which, of course, is one of the principal advantages of Wikipedia. In a system where anyone is free to get something started, however badly, a short, uninformative article can be the anchor for the good article that will eventually appear. Its very inadequacy motivates people to improve it”

In short, something is better than nothing, and there is never really an excuse not to at least start a “stub.” Then if someone is unhappy with the stub at some point, they can improve it. The hardest step for many people, actually starting something, is already done.

In system administration, stealing from Tom Limoncelli, I think the best “stub” is usually a checklist for most system administration stuff. So for example I just started one for “Deployment Steps” which doesn’t actually include how to do anything — just what to do. If someone wants to, they can easily add to it later. Other methods include spreadsheets in Google Docs for licenses and IP address lists, and very short “meeting minutes” which just include decisions made at the meeting and things people said they would do.

On the personal side of things, I have started to use the Inbox Zero technique and Remember the Milk for my checklists. Both of these help me keep my head clear and “Inbox Zero” helps me make sure I don’t miss emails. I have also started to try to batch my interruptions by checking email and chat every 20-30 minutes instead of constantly for at least part of the day.

It doesn’t matter if these are the ultimate-super-top-of-line-over-engineered organizational systems, what matters is that they are simple enough to use and get started with so we don’t pass the point of no return.

(P.S. Before you point out all the obvious flaws in my analogy and that I’m mutilating popular physics, please read Miguel de Icaza’s Well, Actually)

For most of us in IT, when we want to buy some equipment or hardware, we have to get it approved by someone. When you go to this person and say you want to buy a new server, they often want to know:

  1. How much it costs
  2. What is the benefit

The evaluation method is simple, if the costs are less than the benefits (profit) then you buy it, if the costs are more, you don’t buy it. What this ends up being is an effective way to cut short term costs. In the long run though, it will often end up costing more. The reason is that with the way people practice this, they end up making the wrong decision. One reason is that technical people often are not that good at explaining themselves. But more importantly, there are inherit limits in this sort of thinking. Why?

People know more than they can say

I once learned about an experiment that demonstrated that people’s hearing was more sensitive than people originally thought. The traditional experiment was to have people tap if they heard sound, and not tap if they didn’t hear the sound as the sounds were made quieter. From the results of doing this with a lot of people, you could find out where people could no longer recognize sound, right?

Wrong. Eventually someone came along and changed the experiment. The person was no longer supposed to tap when they heard a sound, but rather this time they were to tap when they guessed that they might have heard a sound. The result was that people would “guess” right almost all of the time at levels that were previously believed to be out of the range of hearing.

I went to college for music, and a thing in music that only the “special” people have is the ability to sing a note from memory without having heard it in a while. Most people couldn’t do this. I had a teacher however, that insisted that most people could and just didn’t know it. Again, when they were asked just to “guess”, they were amazed that they got it right most of the time.

The lesson I gathered from both of these things was that people’s instinct, things they know but can’t quite say for sure, has actual value. We know things that we don’t even realize we know.

Case Study: Buying more servers than you might need

Take a common example in IT, and that is buying more servers. The traditional cost benefit analysis is probably includes things like:

  • The cost of not having enough hardware for your application and becoming a little slow.
  • The cost of going down if the additional server is for redundancy.
  • The depreciation of the hardware. This is an important one, hardware loses value fast, if you didn’t actually need it, by the time you do it will probably be a lot cheaper by then.

This sort of thinking is what I have usually seen in IT. There may be more variables, but they are always things that are very concrete and easy for people to comprehend and assign values to. The cost of downtime, cost of the site being slow, etc.

But they leave out the most important category of benefits, the ones that make up the awesome factor.

The Awesome Factor

When George or I ask Jeff about our budget for something, sometimes he says “Just Make it Awesome.” If you are used to traditional IT, this seems a little bit silly.

It’s not.

Going back to our case study, let’s think about some things we didn’t account for:

Pride. Pride is what people get when they make something that is awesome. When you have everything you need to make something awesome you will spend more time on it and do it right. If you don’t have the hardware to set up the redundancy or get great performance, people will care less.

Momentum. Lets pretend the developers just came up with a great feature, it requires more resources, but they were so excited they just programmed it over the weekend. When it comes time to push it next week, and IT tells them they can’t push it to production until a new server gets approved, ordered, setup, tested, and then deployed, they will get discouraged. Eventually, they won’t even think about new features.

Inspiration. Having good tools is inspiring, they give you new ideas on how to do things better and they are fun to learn. Not having what you need is just frustrating.

So, what is the dollar value of pride, momentum, and inspiration? In other words, what is the dollar value of being awesome and just how do you fit that into a cost-benefit analysis. I guess it is possible, you can look at turnover rates in your employees etc, but people just don’t work this way. Creativity in work comes from intuition — not the sum of a bunch of tangible factors.

In computer operations scalability is not about:

  • How many servers you have
  • How fast your hardware is
  • How many data centers you have
  • How much traffic you have
  • Crude and obvious double entendres

Rather is about a mindset.

Not just Google

I recently spoke at PICC with George and took the opportunity to listen to some talks there as well. Tom Limoncelli from Google spoke about some of his thoughts on a university level degree for system administrators. One of the ideas he presented was that the “future of system administration is going to be less about support and more about scalability.” When he said that, someone blurted out what I imagine a lot of us were thinking:

“Of course you think that, you work at Google!”

His answer was something along the lines of,

“Well that really doesn’t matter, even when I was starting out and was a lone system administrator, I still had to scale my time.”

This short dialogue captured the essence of what I think it means to be scalable, and that is doing more with less.

Why do people think it is about size?

When a web site starts to get more traffic and the things that usually follow (More servers, more people, etc) then they basically have no choice but to figure out how to do more with less. This is really for two reasons.

The first reason is that system itself might just break down because it is the wrong way to do it. Sam Saffron put this perfectly when he was interviewed at MIX: “By adding more servers all we would really be doing is distributing the slow”. (Why is turning non-nouns into nouns so catchy?). The other reason is that the cost to throw more hardware at the problem starts to become untenable.

When faced with these problems, a company will generally have no choice but to learn how to scale. But when they are learning to scale because of this, what they are really learning how to do is to do more with less.

Do more with less

You can start to become scalable very early on, even with only a few servers. There are lots of ways we generally practice scalability, a few examples are:

  • Code and script management tasks. When you have 3 servers you may not need to do this, you can probably do it by hand. However, when you have 100 you will have no choice.
  • Use algorithms and data structures that are efficient.
  • Use caching effectively.
  • Document tasks so you are not a single point of failure and so you don’t have to relearn things every time.
  • Use centralized authentication, configuration management, updates, etc.
  • Use automated building and deployment processes.

There are two traits in all off the items I listed above:

  1. They all save time in the long run (they are asymptotically superior) which results in doing more with less.
  2. You don’t need very many servers to do them.

I won’t deny that there are some unique problems that will only start to show themselves when you get really big, and that only at certain sizes do you discover that some systems will start to break down. However, in reality you only need a small amount of hardware to start practicing the principals of scalability.

Just about every introductory class for algorithms teaches you three primary ways to look at algorithm performance:

  • Worse Case Scenario
  • Average Case Scenario
  • Best Case Scenario (This is always followed up by why this isn’t very useful, but they mention it anyways).

You can use this same sort of analysis when thinking about your web performance. I have found that a histogram captures all 3 of these pretty nicely since it is a distribution. The X axis is the response time in milliseconds of a request, and the Y axis is the number of hits that had that response time . Our HAProxy web logs capture the response time of each request from the perspective of the load balancer. This is a nice perspective because it includes pretty much the full stack of a web request that is directly under our control.

We have started to insert our web logs into SQL server. Raw SQL is hard to beat for deep analysis, but I have also started to build a web front end which is particularly useful for generating graphs easily. With a web interface it is easy to filter on certain criteria and you can easily see the distribution for a particular page or a certain client like Googlebot:

(The X axis is the server response in MS, and the Y axis is number of hits that fell into that response bracket)

A crawler will also be a sort of worst case response time perspective on another level because because crawlers cause more cache misses. Clients will generally perform much differently. For example looking at response time for user agents with “Chrome” in them the response time has a very different shape:

In general, I always find it is best to think about how you are viewing your data, and if it is the best way to summarize what you really care about. Average is useful, but it just isn’t the complete picture.