Look at your infrastructure and the way you handle your operations and ask yourself:

“Are we more like the Borg or the Federation?”

If you are not sure, one way you can tell is to look at the pieces of your infrastructure and see if they have personality. If a server has personality, then it is more like the Enterprise. There may be sister ships, like the USS Yamato, but it has its own personality because Geordi has made improvements and its crew has made it unique. Disgusting.

Why disgusting? Personality is the antithesis of scalability. With a large number of servers, each with their own personality, you will need more administrators, and a chaotic management structure. The Borg are the good guys. This is because of some of the distinct advantages of being more like the Borg:

• In the Borg collective, individual drones and ships can be destroyed without much thought. They are easily replaced.
• There are different types of Borg vessels, cubes, spheres, tactical cubes, but within these types, there is no personality.
• Improvements that are assimilated become part of their whole collective making them highly adaptable

In IT operations, your servers should be drones that are easily replaced. You may have different classes of servers, but if they are all the same within each class they are easier to manage. For example, if you have a different brands of hard drives and RAID cards in each server, you will have spend time figuring out the settings, tweaks and compatibility issues for each of these different types. You will also have to track updates for each of these two different things and figure out methods of deployment for each of these types if a vendor doesn’t provide tools to manage these. With configuration and centralized management you can update and adapt improvements to all your servers rapidly. Deploying a change to all servers is just a script or a policy, not a manual process for each server. If you hand code each configuration file and manually deploy software they will start to become different over time when mistakes are made — this is one of the main ways servers develop personality.

At Stack Exchange, we have done a good job at achieving this with our web tier. We do have 1 staging web server, but for our other 9 web servers, the seventh is just seven of nine. George has made a deployment process which includes everything we need, making any web server disposible, and a new one ready for assimilation. Should one of them be destroyed, the others will automatically take over without concern. However, in some areas we are still more like the Federation. We have 4 database servers, and we have noticed areas where our db01/db02 pair is unlike our db03/db04 pair. If one of our primary servers fail, we will mourn the loss as we carry out a manual fail over process. If we were more like the Borg in this area, we could initiate self destruct on one of these servers without a care as the Borg queen would do.

The Borg are the role model, not the Federation. Next time you look at a server that has personality, your first thought should be: “You will be assimilated.”

At a recent talk at PICC, part of my introduction was:

“Hello and thank you for coming to our talk. This is George Beech and I am Kyle Brandt, and we are the two sysadmins at Stack Exchange…”

While “two” is the technically correct answer going by our self-anointed titles, it isn’t really the whole picture.

The Sysadmin Continuum

In reality here at Stack Exchange we all wear a lot of hats. While George and I may own the system administration tasks, we work together with our developers who are pretty accomplished sysadmins in their own right. If it were just George and I soely doing the work that could be considered system administrator work, we would probably have a pretty tough time of it at only two people.

In reality we are continuum. System administration is just a spot on the continuum where George and I spend most of our time. However, a lot of our developers spend a lot of time there and travel the same road.

Some recent examples:

• Jarrod Dixon worked with me on developing a log daemon for inserting HAProxy syslog data into SQL Server
• Nick Craver frequently hops on Dell and makes hardware recommendations
• Jeff Atwood helps us with SSD research
• Sam Saffron works on SQL restore scripts
• Geoff Dalgas racks servers out in OR and troubleshoots our NAS
• Ben Dumke and Jeff Atwood do some CDN research
• Jeff Szczepanski, our Vice President of Products, analyzes packet dumps

Because of the spirit of shared responsibility, we get to take advantage of everyone’s knowledge and experience and we are all better for it.

So in the end, “How Many System Administrators does Stack Exchange Have?” is a question I can’t really answer.

In the life of a company and in the life of a person there is a line that gets crossed when it comes to being organized. When you start out in a business basic documentation and good organizational systems are often optional.

To be clear, I am not saying that they should be optional, rather, that from my experience this is just the reality of the situation. There is usually only a couple of people working on something, so they can always ask each other things and the amount information is small enough that people can mostly keep it in their heads. Also, to be more clear, I don’t really think this is the best way to do things, by its very nature it is inefficient. But in a fast moving startup, it is easy to give into the temptation to skip some of the niceties and get to the next project.

But then one day, you start to get close to the Organizational Event Horizon or OEH. The way you will realize it is by the way it feels, it is the feeling of being Spaghettified. In other words, you get stretched beyond what you can handle. Before you cross the OEH, good organization is about being more efficient and getting more things done. But once you have crossed the OEH, organization becomes the difference between success and failure. Important and critical tasks will start to get missed, and the stress of all the different things going on will tear you apart. If you don’t start to deal with the situation before crossing the OEH, failure in inevitable.

How To Handle It?

I feel like at least on the sysadmin things here at Stack Exchange I am starting to feel the pull. To fix this I think the key is to implement a few basic systems for organization that are based on two related principles: The KISS (Keep it simple stupid) and the “stub” principle. The idea behind both of these is to have the minimal amount of resistance to actually start using documentation and organizational systems. The KISS principle is well known. What I call the “stub principle” is something I picked up from wikipedia and Clay Shirky’s Here Comes Everybody:

“…as long as the experts did nothing (which, on Nupedia, is mostly what they did), nothing happened. In an expert-driven system, an article on asphalt that read “Asphalt is a material used for road coverings” would never appear, even as a stub. So short! So uninformative! Why, anyone could have written that! Which, of course, is one of the principal advantages of Wikipedia. In a system where anyone is free to get something started, however badly, a short, uninformative article can be the anchor for the good article that will eventually appear. Its very inadequacy motivates people to improve it”

In short, something is better than nothing, and there is never really an excuse not to at least start a “stub.” Then if someone is unhappy with the stub at some point, they can improve it. The hardest step for many people, actually starting something, is already done.

In system administration, stealing from Tom Limoncelli, I think the best “stub” is usually a checklist for most system administration stuff. So for example I just started one for “Deployment Steps” which doesn’t actually include how to do anything — just what to do. If someone wants to, they can easily add to it later. Other methods include spreadsheets in Google Docs for licenses and IP address lists, and very short “meeting minutes” which just include decisions made at the meeting and things people said they would do.

On the personal side of things, I have started to use the Inbox Zero technique and Remember the Milk for my checklists. Both of these help me keep my head clear and “Inbox Zero” helps me make sure I don’t miss emails. I have also started to try to batch my interruptions by checking email and chat every 20-30 minutes instead of constantly for at least part of the day.

It doesn’t matter if these are the ultimate-super-top-of-line-over-engineered organizational systems, what matters is that they are simple enough to use and get started with so we don’t pass the point of no return.

(P.S. Before you point out all the obvious flaws in my analogy and that I’m mutilating popular physics, please read Miguel de Icaza’s Well, Actually)

For most of us in IT, when we want to buy some equipment or hardware, we have to get it approved by someone. When you go to this person and say you want to buy a new server, they often want to know:

1. How much it costs
2. What is the benefit

The evaluation method is simple, if the costs are less than the benefits (profit) then you buy it, if the costs are more, you don’t buy it. What this ends up being is an effective way to cut short term costs. In the long run though, it will often end up costing more. The reason is that with the way people practice this, they end up making the wrong decision. One reason is that technical people often are not that good at explaining themselves. But more importantly, there are inherit limits in this sort of thinking. Why?

People know more than they can say

I once learned about an experiment that demonstrated that people’s hearing was more sensitive than people originally thought. The traditional experiment was to have people tap if they heard sound, and not tap if they didn’t hear the sound as the sounds were made quieter. From the results of doing this with a lot of people, you could find out where people could no longer recognize sound, right?

Wrong. Eventually someone came along and changed the experiment. The person was no longer supposed to tap when they heard a sound, but rather this time they were to tap when they guessed that they might have heard a sound. The result was that people would “guess” right almost all of the time at levels that were previously believed to be out of the range of hearing.

I went to college for music, and a thing in music that only the “special” people have is the ability to sing a note from memory without having heard it in a while. Most people couldn’t do this. I had a teacher however, that insisted that most people could and just didn’t know it. Again, when they were asked just to “guess”, they were amazed that they got it right most of the time.

The lesson I gathered from both of these things was that people’s instinct, things they know but can’t quite say for sure, has actual value. We know things that we don’t even realize we know.

Case Study: Buying more servers than you might need

Take a common example in IT, and that is buying more servers. The traditional cost benefit analysis is probably includes things like:

• The cost of not having enough hardware for your application and becoming a little slow.
• The cost of going down if the additional server is for redundancy.
• The depreciation of the hardware. This is an important one, hardware loses value fast, if you didn’t actually need it, by the time you do it will probably be a lot cheaper by then.

This sort of thinking is what I have usually seen in IT. There may be more variables, but they are always things that are very concrete and easy for people to comprehend and assign values to. The cost of downtime, cost of the site being slow, etc.

But they leave out the most important category of benefits, the ones that make up the awesome factor.

The Awesome Factor

When George or I ask Jeff about our budget for something, sometimes he says “Just Make it Awesome.” If you are used to traditional IT, this seems a little bit silly.

It’s not.

Going back to our case study, let’s think about some things we didn’t account for:

Pride. Pride is what people get when they make something that is awesome. When you have everything you need to make something awesome you will spend more time on it and do it right. If you don’t have the hardware to set up the redundancy or get great performance, people will care less.

Momentum. Lets pretend the developers just came up with a great feature, it requires more resources, but they were so excited they just programmed it over the weekend. When it comes time to push it next week, and IT tells them they can’t push it to production until a new server gets approved, ordered, setup, tested, and then deployed, they will get discouraged. Eventually, they won’t even think about new features.

Inspiration. Having good tools is inspiring, they give you new ideas on how to do things better and they are fun to learn. Not having what you need is just frustrating.

So, what is the dollar value of pride, momentum, and inspiration? In other words, what is the dollar value of being awesome and just how do you fit that into a cost-benefit analysis. I guess it is possible, you can look at turnover rates in your employees etc, but people just don’t work this way. Creativity in work comes from intuition — not the sum of a bunch of tangible factors.

In computer operations scalability is not about:

• How many servers you have
• How fast your hardware is
• How many data centers you have
• How much traffic you have
• Crude and obvious double entendres

Rather is about a mindset.

Not just Google

I recently spoke at PICC with George and took the opportunity to listen to some talks there as well. Tom Limoncelli from Google spoke about some of his thoughts on a university level degree for system administrators. One of the ideas he presented was that the “future of system administration is going to be less about support and more about scalability.” When he said that, someone blurted out what I imagine a lot of us were thinking:

“Of course you think that, you work at Google!”

His answer was something along the lines of,

“Well that really doesn’t matter, even when I was starting out and was a lone system administrator, I still had to scale my time.”

This short dialogue captured the essence of what I think it means to be scalable, and that is doing more with less.

Why do people think it is about size?

When a web site starts to get more traffic and the things that usually follow (More servers, more people, etc) then they basically have no choice but to figure out how to do more with less. This is really for two reasons.

The first reason is that system itself might just break down because it is the wrong way to do it. Sam Saffron put this perfectly when he was interviewed at MIX: “By adding more servers all we would really be doing is distributing the slow”. (Why is turning non-nouns into nouns so catchy?). The other reason is that the cost to throw more hardware at the problem starts to become untenable.

When faced with these problems, a company will generally have no choice but to learn how to scale. But when they are learning to scale because of this, what they are really learning how to do is to do more with less.

Do more with less

You can start to become scalable very early on, even with only a few servers. There are lots of ways we generally practice scalability, a few examples are:

• Code and script management tasks. When you have 3 servers you may not need to do this, you can probably do it by hand. However, when you have 100 you will have no choice.
• Use algorithms and data structures that are efficient.
• Use caching effectively.
• Document tasks so you are not a single point of failure and so you don’t have to relearn things every time.
• Use centralized authentication, configuration management, updates, etc.
• Use automated building and deployment processes.

There are two traits in all off the items I listed above:

1. They all save time in the long run (they are asymptotically superior) which results in doing more with less.
2. You don’t need very many servers to do them.

I won’t deny that there are some unique problems that will only start to show themselves when you get really big, and that only at certain sizes do you discover that some systems will start to break down. However, in reality you only need a small amount of hardware to start practicing the principals of scalability.

Traffic statistics and numbers are fun. If you haven’t noticed, we like to post our stats on our blogs and places like Hacker News. However, I have noticed a trend in comments, and that is that people use our numbers and compare them to numbers of some other site using a different stack (often some variation on the LAMP stack).

This just doesn’t make sense. There is really no way to compare the best case performance of two different stacks such as comparing the Microsoft stack to the LAMP stack for dynamic applications. To illustrate this, let us think about what might be required in an experiment that might compare these two stacks in a way that might actually be meaningful.

Things we would need:

• Two teams of programmers and system administrators. You would have a different team for each stack, and the teams would need to be made up of experts on their stack.
• A highly detailed spec about an application that is reasonably feature rich. Each team would need to develop this application on their stack for probably at least a year.
• Enough users using both versions of the application such that people would consider this to be pretty high traffic site over a long period of time. You also would need enough users that possible differences in user behavior would no longer be significant.

Getting the above is never really going to happen, and even if did the technologies probably will have changed by the time the results would be published. Also, even with the things I listed the experiment is pretty weak without many repetitions of the experiment with different teams.

So, why do we post are numbers and do they even mean anything? Well, like I said, they are just fun. However, there is some meaning from our example. Our example shows that it is possible to use the Microsoft stack to do something similar to what we do with the amount of traffic we get with a good team. The scope of this conclusion is limited, but it is also potentially useful to people which is one reason we like to be open about our numbers and our operations.

When it comes to system administration there are a two main questions that come up in relation to RAM:

• Which and how much RAM do I buy?
• How do I properly install it?

In order to be able to figure this out there are some fundamentals regarding RAM which a sysadmin has to know for servers. To be practical lets figure out what a particular stick of RAM is about (Which happens to be in our database servers):

> Registered ECC Dual Rank 8GB DDR3-1333 (PC3-10600)

Frequency and Bandwidth:

Wikipedia tells us that “frequency is the number of occurrences of a repeating event per a unit of time.” In the case of memory the “repeating event” we are concerned with is bits of data being sent from the cpu to RAM and the “unit of time” is a second. With memory this is done so many times we measure it in millions of times per second (aka megahertz or MHz). The bits are chunks of 64 bits which are often referred to as “words”. So the transfer frequency of our example RAM is 1333 MHz (the clock frequency is different, more on this later).

So how “fast” is this? In computing, speed is often measured in how much data you move in a certain amount of time (There is also latency). A common measurement is the amount of megabytes in a second. There are 8 bits per byte. We said before that this RAM can push 64 bits 1,333 million times per second. So 64 * 1,333,000,000 / 8 gives us 10,664,000,000 bytes a second, or roughly 10,600 MegaBytes per second. So we now have our bandwidth, 10,600 MB/s:

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

DDR3:

DDR stands for Double Data Rate. In the case of DDR memory the data transfer event actually manages to carry two chunks of data. With DDR2, you get 4, and with DDR3 you get 8. This does not include the rate of operations, but rather the amount of data chunks carried in each operation. What we end up having is different frequencies going on — clock ticks from the memory controller (which correspond to operations), bus and cpu frequencies, and finally an effective transfer frequency. These frequencies are all related to each other since this memory is synchronous. At the heart of all of this is the system clock, but memory is labeled by the transfer frequency because higher numbers sell better and it is probably the number most people would be interested in anyways. So the 10,600 MB/s throughput is based off of the transfer frequency and therefore takes DDR3 into account.

When it comes to “PC3″ I am not entirely sure what that stands for, but DDR2 = PC2 and DDR3 = PC3. I am hoping a reader knows what it is for sure and can back it with some evidence.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Ranks and Capacity:

A rank is a independently accessible 64-bit area of memory. Today you buy memory in single, double, and quad ranks. To really get into ranks you will need to understand the basic signals used in ram which you can find in the “What every programmer should know about memory” in the referecences and then there is some detail on ranks in Addressing Increased Capacity Demand Using Commodity Memories. As a system administrator the main thing to understand is that there may be limitations on your motherboard such as the ability have mixed ranks and where certain ranks can go (On the R710, Quad-Ranks must go in the first DIMM slot of each channel).

The capacity is “8GB” — simply how much memory there is on the chip.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Registered and ECC

Registered RAM is also referred to as buffered RAM. It has a buffer between the memory controller and the memory chips. The end result is that you can have a higher capacity of RAM in the system but there is a little more latency. Memory you buy for servers will generally be registered memory. ECC stands for Error Correcting Code and includes parity on the memory to help ensure that the memory operation is valid.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Architecture, Channels, and the i7:

In the frequency and bandwidth section I mentioned that 64 bit chunks are being sent at a time. These chunks are being sent over a channel. When you have multiple channels the basic idea is that you are increasing your width — so dual channel is chunks of 128 bits. The current Intel i7 architecture supports 3 channels per processor (as in a chip you would hold in your hand). So in a dual processor system you have up to six channels. So with our simplified model we are talking rougly 30 GB/s of memory throughput per processor (10,600 * 3).

With recent architectures the memory was connected to the processor via the Front Side Bus (FSB). However with the i7 the memory controller is on the processors themselves and the memory is directly connected. There is also QPI (Quick Path Interconnect) which connects the PCI express bus to the processor. In multiple processor systems it also connects the processors together, so if Processor B wants Processor’s A memory it goes over the QPI.

There is a lot more to learn about memory. See the references section at the end of this post. Reading and writing about memory is difficult because there are a lot of differences between the general theory and what is actually happening so I and others skip over details. Reading the references or at least parts of them is probably needed to have a good picture of memory functions. You might also want to read about signal timings and interleaving, but I think the above covers the basics for a generalist system administrator.

Trade Offs — Channels, Capacity, and Speed:

So why does all this matter to a system administrator? The fundamental answer is that there are tradeoffs. Besides cost you have memory speed, error handling, and total capacity tradeoffs in the configuration you choose.

In our current database server, Dell R710, we are using the Intel 5600 series architecture. We have 2 processors which each have 3 channels of memory directly connected to each processor for a total of 6 channels. There are three general RAM population choices (Reference: Dell Poweredge 11g Whitepaper):

1. Optimize for speed (throughput). This configuration does not give the most RAM in the system in terms of capacity, but does give the fastest throughput possible for the memory. For the Dell R710, 4GB Registered DIMMs achieve this with a maximum amount of memory of 12 GB per CPU. This will let you run your memory at 1333 MHz.
2. A balance of speed and capacity. For this you would want dual-rank 1066 GB registered DIMMs. This allows for a capacity of 48GB per CPU but the memory can be clocked at 1033 MHz.
3. Optimize for capacity. With this option you would use the same chips as the balanced configuration but they get down clocked to 800 MHz for a total of 144 GB capacity.

According to Intel you get the best speed out of a balanced three channel configuration at this capacity. We just upgraded to 96 GB from 64 GB and have gone from a 2 channel configuration to a 3 channel configuration. This upgrade requires that the memory gets down clocked from DDR3-1333 to DDR3-1033. Although the clock speed has decreased for our memory, the number of channels or width has increased along with our capacity. Doing the math this in theory yields more memory throughput:

Two Channel:
(128 * 1,333,000,000) / 8 = 21,328 MB/s
Three Channel:
(192 * 1,033,000,000) / 8 = 24,792 MB/s

So our 3 channel configuration in our server is as follows:

So we match exactly option number 2 currently. Part of the reason you buy from a vendor like Dell is that they should be helping you get the ideal configuration, but knowing your RAM at a basic level will help make sure you are getting what you need.

Update:

George had informed me that he saw a memory speed 1333 MHz at boot. After looking at the output of CPU-Z and seeing that his was confirmed there as well I became confused as to how we could have 1333 MHz at a 96GB capacity:

After speaking with Dell I found that there was an information update regarding system memory for the R710. For 2R DIMMs they clock at 1333 not 1066 when you have 2 DIMMs per channel. So the proper calculation for our configuration is

(192 * 1,333,000,000) / 8 = 31,992 MB/s

This means that there was no trade off in clock speed when upgrading our capacity and to three channels.

References:

Kingston Technology memory ranking technical brief
What every programmer should know about memory — Highly Recommend, for those that want a lot of detail (PDF Version).
Memory technology evolution: an overview of system memory technologies, 8th ed.

The standard way to measure uptime is to measure the percent of time that your application is available over a period of time. This is always a good idea to do, but I don’t think it is the best way to measure the availability of your application for many companies. I think a better primary way is to think about how confident you are, your coworkers, and customers are in the availably of your application. If everyone is confident that your products will be up pretty much all the time then I think that this is a better measure than any “nines”.

I admit that going on how you feel as a primary measurement may seem a bit silly or touchy-feely. But I think your feelings and intuition solves a lot of issues with measuring uptime naturally. When even trying to figure out what a good uptime target is in percent of uptime some things you have to consider are:

Do you include maintenance time? This is probably more of an SLA issue than it is a practical one. For some sites, being down during off hours really isn’t a big deal. As an example, imagine an application that serves grade schoolers in the United States — it really doesn’t matter if it is down in the middle of the night.

Financial cost of achieving uptime vs cost of downtime? Some companies can map this pretty easily — “we loose X dollars per 30 seconds of downtime”. You then figure out the man power and equipment cost of high availability and all you need is some basic math skills. However, lots of companies are not finance or commerce companies so this is more difficult to measure. Take stackoverflow.com for example. We do have advertisers and our careers product, but our most valuable factor in my opinion is our users and our communities. How do you measure this other than keeping close ties to your users and see how they feel?

Development speed vs uptime? It seems intuitive that a fast development cycle will lead to more interruptions than a more careful and tested process. This might not be true, faster release cycles often mean smaller changes and might in the end cause less problems. I am not qualified to answer this, but it is another factor in uptime. If we accept this as true (I don’t, personally), then you have to weigh the cost of being down vs new features. The way this cost is measured is going to be difficult.

What counts as being down? If the site is slow, is this the same as being down? For example, at a certain point you might lose a customer to another commerce site if it is taking forever to load the purchase page. This is related to how Google now includes latency of your site in their page rank system. If you have new serious bugs but the site is still up, is this basically the same as being down?

Recovery Time: I think one of the most important things in uptime is recovery time. The primary thought most people have is “How do I keep from going down?”. Things go down no matter what, Facebook and Google have had some uptime issues before and everyone will. However, how much downtime you have when you do go down is a matter of how fast you can recover. I think recovery speed is a better place to start than attempting to eliminate going down at all. Of course, you need to do both.

The CHI of uptime: So if you are going to capture availability in a number it probably needs to include the above as factors in your equation as well as other things I probably haven’t even thought of. I heard a talk by Dharmesh Shah at Business of Software 2010 about measuring his customer’s happiness by factoring in various measurable stats as a Customer Happiness Index (CHI). (You can read about this talk here). Trying to do this with uptime would need to capture the above issues and is a worthy goal. However, getting there is not a small goal either.

Getting a good feeling for how confident your customers and employees are about the availability of your application captures all of these issues naturally. Uptime stats have their value, but I think numbers and stats are often overrated.