On more than one occasion I have been asked “how do I get a job in IT?” This question could easily be shrugged off and relegated to the canned answers pile; phrases I have personally uttered in the past include “go get some certifications then send out resumes” or “play with computers for a few years and then apply.” These answers really are a cop-out on my part so I’d like to take a few minutes to apply some serious thought to the query and offer my thoughts.

What People Will Expect of You

While most people would say that having a large breadth and depth of knowledge in computers is the primary requirement for being employed in IT, I would disagree with this. There are a handful of very critical attributes that you will find in any IT Rockstar:

  • Good Problem Solving Skills
  • Critical Thinking
  • Strong work ethic
  • Ability to handle “Burst Stress”

Problem Solving is possibly the most important attribute any good IT team member will have. Problem solving is the core of our vocation, when you think about it. IT is tasked with solving problems that the other employees aren’t able to handle. You’ll also find that in many workplaces, the IT guys are the ones helping to solve other complex problems in the enterprise because of the problem solving skillset they demonstrate. Critical Thinking is another important skill to have when talking about technical jobs. Oftentimes the problem I’m trying to solve does not have an easy-to-see solution (sometimes, the cause itself is not immediately apparent.)

Do you like working long hours for little praise or thanks? That’s pretty much working IT in a nutshell. Performing server maintenance usually needs to wait until after hours, which generally means you’ll be putting in a full 8 hours for that workday, then many more hours that night to do the after-hours downtimes. Couple this with “burst stress” and you’ve got a recipe for gray hair. Burst Stress is common in jobs like police work or firefighting, where you have long periods of low stress followed by short bursts of extremely high stress situations.

What do you want to do?

There are a handful of disciplines in IT, each with their separate purposes. What follows is a breakdown of the individual disciplines and what that type of work entails. Regardless of what path you choose, you’ll likely start in an IT Generalist role, and many people stay in that role their entire career. I myself genuinely enjoy being a generalist; it means that any given day I might be working on any number of problems that aren’t the same old issues over and over again.

IT Generalist — This is often the least respected but most useful role in IT. As a generalist, you are a Jack of All Trades. You’re expected to understand not only Desktop and Server Support, but also have a useable knowledge set in Telephony, Networking, Backup and Disaster Recovery, and Security. When you start your new IT career, you’ll usually start as a quasi-apprentice in a generalist role. Once you’ve “earned your stripes” in a few disciplines, you’ll be more educated and ready to move to a specialization, if that’s your pleasure.

Desktop Support — The Desktop Support team will generally be tasked with maintaining the software and equipment that other people in the company utilize to get their jobs done. This job is very demanding from a customer service perspective, as the person doing this job will oftentimes be assisting users who are already annoyed that their machines aren’t working the way they’d like! Skillset wise, Desktop engineers need to be highly fluent in whichever desktop operating system your company uses, as well as any applications the company makes use of. Generally speaking, you’re going to need to know Windows and Office like the back of your hand if you want to get a job like this in the majority of businesses.

Server Support/Systems — The Systems or “Server Support” team has similar requirements as Desktop Support when it comes to strong knowledge sets of operating systems and software. The jobs diverge when you look at the “back-of-the-house” operations that a Systems team is often assigned. Knowledge of datacenter operations (power distribution, hot row/cold row, rack positioning) is essential in this role if your organization has more than just a “closet in the back with a few servers.” Server Support is also usually in charge of all backend/utility systems in the organization, including directory authentication, mail and groupware, and administrating backups.

Networking — This group focuses mostly on interconnecting sites and equipment together. To get into this group, you better be prepared to prove you know the OSI model like the back of your hand. This is a tough group to coast in; TCP/IP can be fickle under wavering hands. Usually, a particular company will standardize on one vendor for their equipment, and you’ll be expected to know the operating system for that particular vendor. If you’re looking to get into the networking team at a new company, you should likely know Cisco and Juniper OS’s (Cisco IOS/ASA-OS, JunOS, etc.) Even if the company does not use either of these major brands, having knowledge of how other platforms operate can assist you in doing the job. Networking also is susceptible to the above mentioned “Burst Stress”, since if a site-to-site link is down, it’s possible that hundreds of people are sitting waiting for you to fix the problem.

Security — To be a member of the security team is to be alternately loved and hated by differing groups in your organization, often different groups at different times. Security generally finds themselves tasked with doing audits of internal data security and setting standards to help achieve compliance with data security standards. Computer forensic applications are the toolbox of the IT Security team; to do the job it will help to be seasoned at data analysis/data mining and event correlation. Security is usually the team tasked with defending audits as well, so it helps if you have good verbal communication skills. Finally, Security is sometimes tasked with coming up with the Disaster Recovery and Business Continuity Plans for a business. In some cases, other departments handle this, but in my experience this has been a Security-type role.

Database Administration — DBA is a tough specialization to be hired in. Brent Ozar has a great blog post covering this, so I won’t go in to too much detail but instead refer you to Brent Ozar’s excellent blog post series linked here. Suffice to say, you need to be able to speak SQL and make it sound like Shakespeare before you can roll with the titans like Mr. Ozar.

Telephony — I’m hesitant to group Telephony into IT, but many times it falls under the moniker of IT, so we’ll cover it here. Telephony is the specialization that handles telephones and telephony technologies. You’re going to want to know the details of how signaling works in PSTN networks, things like TDM, FXO,FXS, E&M, CAS, and a whole lot of other crazy acronyms. Depending on the size of the organization, you might also need to have some cross-pollination with the Networking skillset.

In a future blog post, I will cover what I consider the “elephant in the corner” when it comes to hiring in IT. Remember, it’s not always what you know, but who you know that gets you hired. As always, your comments and criticisms are welcome/encouraged; I’d like to hear what the community feels on this topic.

System administrators can sometimes seem like they are trying to get everyone down. Someone goes to them with a great idea, application, feature, or just something they just want to get done. When the sysadmin comes in, instead of being enthusiastic about the great idea, he starts talking about viruses, hacking, earthquakes, floods, fires, Vogon invasions, and all the things that can go wrong. Because of this, the natural reaction is to stick the sysadmin in the basement and stop inviting them to parties.

In order to work past this issue, it takes a little work from both sides of the table. Coworkers need to remember that disaster is part of a sysadmin’s job and that Murphy’s law is part of what they do. Part of a sysadmin’s job to think about this stuff all the time because we are expected to have backups and to be able to recover from small events to major disasters. Major disasters are uncommon but do happen, hence the importance of offsite backups at a minimum. Small disasters, such as power failures, router failures are actually quite common. When a sysadmin is doing a really good job, people don’t even know about some of the smaller failures.

It also falls on the sysadmin’s shoulders not get stuck in this gloom and doom world. We have remind ourselves not to get in the habit of saying No to everything because something could go wrong. This is important because the danger is that that people will try to start bypassing you or just not believing you when you talk about very real threats because they think you are crying wolf. It largely comes down to being mindful of the bad things that can happen while being enthusiastic about all the good things that can come from a new idea or feature.

It can be tough not be the guy in the street with the end of the world sign when in a way it is part of your job, but learning how to find the balance is one of the challenges of being a sysadmin.

Transfer rates and the number of packets you send are measured in units of a certain quantity of data per units of time. The unit of time that everyone is used to is the second. The standard quantity of data that is used in the networking field is bits and the standard time unit is seconds. So for example, the standard network interface these days is 1 Gigabit per second. So the quantity of data is a Gigabit, and the unit of time is a second. We call this the transfer rate. The key thing to remember is that this is a fixed ratio of data over time. Because of this, you can divide the ratio by any number you want to (Ignoring the complexities of the discrete properties of Ethernet frequencies, system clocking, etc). So, 500 Mbit over a half second is the same fixed ratio as 1 Gigabit per second.

The thing is though, in computing, a second is a really, really, really long time. This is important, because when we choose what unit of time to express this in, what we are doing is graph smoothing (It is sort of, although not really, like taking an average).

For example, we could transfer 900 Mbit in half of a second and another 100 Mbit for the other half of that second. How much data was transferred during that second? The answer is 1 Gbit. If we transfer 500 Mbit per half second and another 500 Mbit per the other half second — this is also 1 Gbit per second:. This effect is illustrated in these Megabits per half second graphs:

These two are clearly not the same thing, but when you express them as the amount of data transfered over a second they are. This is important because a 1 Gbit per second interface is also a 500 Mbit per half second interface — and a 500 Mbit per half second interface can’t transfer 900 Mbits per half second (I am ignoring any buffering effects, but in practice we have found this to be essentially true).

This effect is made even worse by most monitoring tools because most take samples every 5 minutes. So what you are really seeing is the transfer rate per 5 minutes converted to a per second rate. This sort of thing is why people say data can lie.

Why Should you Care?

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

We have bonded these interfaces using Intel Load Balancing (ALB/RLB) and for the most part our discards have gone away. We did this on all but one of our web servers for a while and found that the one that didn’t have the bonded interface had discards climbing while the others did not.

A second is a long time — be wary of trusting it too much to measure things.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

  • When the test was run in UTC time and who ran it
  • Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
  • Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
  • Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.

Look at your infrastructure and the way you handle your operations and ask yourself:

“Are we more like the Borg or the Federation?”

If you are not sure, one way you can tell is to look at the pieces of your infrastructure and see if they have personality. If a server has personality, then it is more like the Enterprise. There may be sister ships, like the USS Yamato, but it has its own personality because Geordi has made improvements and its crew has made it unique. Disgusting.

Why disgusting? Personality is the antithesis of scalability. With a large number of servers, each with their own personality, you will need more administrators, and a chaotic management structure. The Borg are the good guys. This is because of some of the distinct advantages of being more like the Borg:

  • In the Borg collective, individual drones and ships can be destroyed without much thought. They are easily replaced.
  • There are different types of Borg vessels, cubes, spheres, tactical cubes, but within these types, there is no personality.
  • Improvements that are assimilated become part of their whole collective making them highly adaptable

In IT operations, your servers should be drones that are easily replaced. You may have different classes of servers, but if they are all the same within each class they are easier to manage. For example, if you have a different brands of hard drives and RAID cards in each server, you will have spend time figuring out the settings, tweaks and compatibility issues for each of these different types. You will also have to track updates for each of these two different things and figure out methods of deployment for each of these types if a vendor doesn’t provide tools to manage these. With configuration and centralized management you can update and adapt improvements to all your servers rapidly. Deploying a change to all servers is just a script or a policy, not a manual process for each server. If you hand code each configuration file and manually deploy software they will start to become different over time when mistakes are made — this is one of the main ways servers develop personality.

At Stack Exchange, we have done a good job at achieving this with our web tier. We do have 1 staging web server, but for our other 9 web servers, the seventh is just seven of nine. George has made a deployment process which includes everything we need, making any web server disposible, and a new one ready for assimilation. Should one of them be destroyed, the others will automatically take over without concern. However, in some areas we are still more like the Federation. We have 4 database servers, and we have noticed areas where our db01/db02 pair is unlike our db03/db04 pair. If one of our primary servers fail, we will mourn the loss as we carry out a manual fail over process. If we were more like the Borg in this area, we could initiate self destruct on one of these servers without a care as the Borg queen would do.

The Borg are the role model, not the Federation. Next time you look at a server that has personality, your first thought should be: “You will be assimilated.”

At a recent talk at PICC, part of my introduction was:

“Hello and thank you for coming to our talk. This is George Beech and I am Kyle Brandt, and we are the two sysadmins at Stack Exchange…”

While “two” is the technically correct answer going by our self-anointed titles, it isn’t really the whole picture.

The Sysadmin Continuum

In reality here at Stack Exchange we all wear a lot of hats. While George and I may own the system administration tasks, we work together with our developers who are pretty accomplished sysadmins in their own right. If it were just George and I soely doing the work that could be considered system administrator work, we would probably have a pretty tough time of it at only two people.

In reality we are continuum. System administration is just a spot on the continuum where George and I spend most of our time. However, a lot of our developers spend a lot of time there and travel the same road.

Some recent examples:

  • Jarrod Dixon worked with me on developing a log daemon for inserting HAProxy syslog data into SQL Server
  • Nick Craver frequently hops on Dell and makes hardware recommendations
  • Jeff Atwood helps us with SSD research
  • Sam Saffron works on SQL restore scripts
  • Geoff Dalgas racks servers out in OR and troubleshoots our NAS
  • Ben Dumke and Jeff Atwood do some CDN research
  • Jeff Szczepanski, our Vice President of Products, analyzes packet dumps

Because of the spirit of shared responsibility, we get to take advantage of everyone’s knowledge and experience and we are all better for it.

So in the end, “How Many System Administrators does Stack Exchange Have?” is a question I can’t really answer.

In the life of a company and in the life of a person there is a line that gets crossed when it comes to being organized. When you start out in a business basic documentation and good organizational systems are often optional.

To be clear, I am not saying that they should be optional, rather, that from my experience this is just the reality of the situation. There is usually only a couple of people working on something, so they can always ask each other things and the amount information is small enough that people can mostly keep it in their heads. Also, to be more clear, I don’t really think this is the best way to do things, by its very nature it is inefficient. But in a fast moving startup, it is easy to give into the temptation to skip some of the niceties and get to the next project.

But then one day, you start to get close to the Organizational Event Horizon or OEH. The way you will realize it is by the way it feels, it is the feeling of being Spaghettified. In other words, you get stretched beyond what you can handle. Before you cross the OEH, good organization is about being more efficient and getting more things done. But once you have crossed the OEH, organization becomes the difference between success and failure. Important and critical tasks will start to get missed, and the stress of all the different things going on will tear you apart. If you don’t start to deal with the situation before crossing the OEH, failure in inevitable.

How To Handle It?

I feel like at least on the sysadmin things here at Stack Exchange I am starting to feel the pull. To fix this I think the key is to implement a few basic systems for organization that are based on two related principles: The KISS (Keep it simple stupid) and the “stub” principle. The idea behind both of these is to have the minimal amount of resistance to actually start using documentation and organizational systems. The KISS principle is well known. What I call the “stub principle” is something I picked up from wikipedia and Clay Shirky’s Here Comes Everybody:

“…as long as the experts did nothing (which, on Nupedia, is mostly what they did), nothing happened. In an expert-driven system, an article on asphalt that read “Asphalt is a material used for road coverings” would never appear, even as a stub. So short! So uninformative! Why, anyone could have written that! Which, of course, is one of the principal advantages of Wikipedia. In a system where anyone is free to get something started, however badly, a short, uninformative article can be the anchor for the good article that will eventually appear. Its very inadequacy motivates people to improve it”

In short, something is better than nothing, and there is never really an excuse not to at least start a “stub.” Then if someone is unhappy with the stub at some point, they can improve it. The hardest step for many people, actually starting something, is already done.

In system administration, stealing from Tom Limoncelli, I think the best “stub” is usually a checklist for most system administration stuff. So for example I just started one for “Deployment Steps” which doesn’t actually include how to do anything — just what to do. If someone wants to, they can easily add to it later. Other methods include spreadsheets in Google Docs for licenses and IP address lists, and very short “meeting minutes” which just include decisions made at the meeting and things people said they would do.

On the personal side of things, I have started to use the Inbox Zero technique and Remember the Milk for my checklists. Both of these help me keep my head clear and “Inbox Zero” helps me make sure I don’t miss emails. I have also started to try to batch my interruptions by checking email and chat every 20-30 minutes instead of constantly for at least part of the day.

It doesn’t matter if these are the ultimate-super-top-of-line-over-engineered organizational systems, what matters is that they are simple enough to use and get started with so we don’t pass the point of no return.

(P.S. Before you point out all the obvious flaws in my analogy and that I’m mutilating popular physics, please read Miguel de Icaza’s Well, Actually)

For most of us in IT, when we want to buy some equipment or hardware, we have to get it approved by someone. When you go to this person and say you want to buy a new server, they often want to know:

  1. How much it costs
  2. What is the benefit

The evaluation method is simple, if the costs are less than the benefits (profit) then you buy it, if the costs are more, you don’t buy it. What this ends up being is an effective way to cut short term costs. In the long run though, it will often end up costing more. The reason is that with the way people practice this, they end up making the wrong decision. One reason is that technical people often are not that good at explaining themselves. But more importantly, there are inherit limits in this sort of thinking. Why?

People know more than they can say

I once learned about an experiment that demonstrated that people’s hearing was more sensitive than people originally thought. The traditional experiment was to have people tap if they heard sound, and not tap if they didn’t hear the sound as the sounds were made quieter. From the results of doing this with a lot of people, you could find out where people could no longer recognize sound, right?

Wrong. Eventually someone came along and changed the experiment. The person was no longer supposed to tap when they heard a sound, but rather this time they were to tap when they guessed that they might have heard a sound. The result was that people would “guess” right almost all of the time at levels that were previously believed to be out of the range of hearing.

I went to college for music, and a thing in music that only the “special” people have is the ability to sing a note from memory without having heard it in a while. Most people couldn’t do this. I had a teacher however, that insisted that most people could and just didn’t know it. Again, when they were asked just to “guess”, they were amazed that they got it right most of the time.

The lesson I gathered from both of these things was that people’s instinct, things they know but can’t quite say for sure, has actual value. We know things that we don’t even realize we know.

Case Study: Buying more servers than you might need

Take a common example in IT, and that is buying more servers. The traditional cost benefit analysis is probably includes things like:

  • The cost of not having enough hardware for your application and becoming a little slow.
  • The cost of going down if the additional server is for redundancy.
  • The depreciation of the hardware. This is an important one, hardware loses value fast, if you didn’t actually need it, by the time you do it will probably be a lot cheaper by then.

This sort of thinking is what I have usually seen in IT. There may be more variables, but they are always things that are very concrete and easy for people to comprehend and assign values to. The cost of downtime, cost of the site being slow, etc.

But they leave out the most important category of benefits, the ones that make up the awesome factor.

The Awesome Factor

When George or I ask Jeff about our budget for something, sometimes he says “Just Make it Awesome.” If you are used to traditional IT, this seems a little bit silly.

It’s not.

Going back to our case study, let’s think about some things we didn’t account for:

Pride. Pride is what people get when they make something that is awesome. When you have everything you need to make something awesome you will spend more time on it and do it right. If you don’t have the hardware to set up the redundancy or get great performance, people will care less.

Momentum. Lets pretend the developers just came up with a great feature, it requires more resources, but they were so excited they just programmed it over the weekend. When it comes time to push it next week, and IT tells them they can’t push it to production until a new server gets approved, ordered, setup, tested, and then deployed, they will get discouraged. Eventually, they won’t even think about new features.

Inspiration. Having good tools is inspiring, they give you new ideas on how to do things better and they are fun to learn. Not having what you need is just frustrating.

So, what is the dollar value of pride, momentum, and inspiration? In other words, what is the dollar value of being awesome and just how do you fit that into a cost-benefit analysis. I guess it is possible, you can look at turnover rates in your employees etc, but people just don’t work this way. Creativity in work comes from intuition — not the sum of a bunch of tangible factors.

In computer operations scalability is not about:

  • How many servers you have
  • How fast your hardware is
  • How many data centers you have
  • How much traffic you have
  • Crude and obvious double entendres

Rather is about a mindset.

Not just Google

I recently spoke at PICC with George and took the opportunity to listen to some talks there as well. Tom Limoncelli from Google spoke about some of his thoughts on a university level degree for system administrators. One of the ideas he presented was that the “future of system administration is going to be less about support and more about scalability.” When he said that, someone blurted out what I imagine a lot of us were thinking:

“Of course you think that, you work at Google!”

His answer was something along the lines of,

“Well that really doesn’t matter, even when I was starting out and was a lone system administrator, I still had to scale my time.”

This short dialogue captured the essence of what I think it means to be scalable, and that is doing more with less.

Why do people think it is about size?

When a web site starts to get more traffic and the things that usually follow (More servers, more people, etc) then they basically have no choice but to figure out how to do more with less. This is really for two reasons.

The first reason is that system itself might just break down because it is the wrong way to do it. Sam Saffron put this perfectly when he was interviewed at MIX: “By adding more servers all we would really be doing is distributing the slow”. (Why is turning non-nouns into nouns so catchy?). The other reason is that the cost to throw more hardware at the problem starts to become untenable.

When faced with these problems, a company will generally have no choice but to learn how to scale. But when they are learning to scale because of this, what they are really learning how to do is to do more with less.

Do more with less

You can start to become scalable very early on, even with only a few servers. There are lots of ways we generally practice scalability, a few examples are:

  • Code and script management tasks. When you have 3 servers you may not need to do this, you can probably do it by hand. However, when you have 100 you will have no choice.
  • Use algorithms and data structures that are efficient.
  • Use caching effectively.
  • Document tasks so you are not a single point of failure and so you don’t have to relearn things every time.
  • Use centralized authentication, configuration management, updates, etc.
  • Use automated building and deployment processes.

There are two traits in all off the items I listed above:

  1. They all save time in the long run (they are asymptotically superior) which results in doing more with less.
  2. You don’t need very many servers to do them.

I won’t deny that there are some unique problems that will only start to show themselves when you get really big, and that only at certain sizes do you discover that some systems will start to break down. However, in reality you only need a small amount of hardware to start practicing the principals of scalability.

Traffic statistics and numbers are fun. If you haven’t noticed, we like to post our stats on our blogs and places like Hacker News. However, I have noticed a trend in comments, and that is that people use our numbers and compare them to numbers of some other site using a different stack (often some variation on the LAMP stack).

This just doesn’t make sense. There is really no way to compare the best case performance of two different stacks such as comparing the Microsoft stack to the LAMP stack for dynamic applications. To illustrate this, let us think about what might be required in an experiment that might compare these two stacks in a way that might actually be meaningful.

Things we would need:

  • Two teams of programmers and system administrators. You would have a different team for each stack, and the teams would need to be made up of experts on their stack.
  • A highly detailed spec about an application that is reasonably feature rich. Each team would need to develop this application on their stack for probably at least a year.
  • Enough users using both versions of the application such that people would consider this to be pretty high traffic site over a long period of time. You also would need enough users that possible differences in user behavior would no longer be significant.

Getting the above is never really going to happen, and even if did the technologies probably will have changed by the time the results would be published. Also, even with the things I listed the experiment is pretty weak without many repetitions of the experiment with different teams.

So, why do we post are numbers and do they even mean anything? Well, like I said, they are just fun. However, there is some meaning from our example. Our example shows that it is possible to use the Microsoft stack to do something similar to what we do with the amount of traffic we get with a good team. The scope of this conclusion is limited, but it is also potentially useful to people which is one reason we like to be open about our numbers and our operations.