The Three Perspectives

In this three part series I am going to explain a three level framework for monitoring your infrastructure. As an overview, the three levels are:

• Micro: “Ground Level”
• Meso: “Day to Day”
• Macro: “Seasonal”

These levels go from a detailed close up view of your environment to a large-scale view. “Ground Level” monitoring is a highly detailed, micro view of your environment. The “Day to Day” view is an ongoing picture of your entire environment. Lastly, what I call “Seasonal” monitoring is a macro perspective of how your environment changes over months or years.

From my experiences with system administrators, anecdotally I would say that most are only doing the meso level with maybe a touch of the micro when problems happen. The meso level is common because this is what tools like Cacti and Nagios handle. This sort of monitoring system is fundamental to day to day operations. However, the other levels are just important for a high performing environment.

I am breaking this into three levels because each level is handled differently and has different characteristics. Because of the different attributes of these levels there are also different tools suited for each type of Monitoring.

Ground Level Monitoring

“Ground Level” or micro monitoring is high resolution monitoring. By this I mean that you take a lot of samples in short periods of time — generally every second or multiple times a second. These tools are often run from the machines themselves. They also return lots of information. You are probably already familiar with many tools you would use for micro monitoring:

• Perfmon
• Sar
• Wireshark / TCPDump
• Web Logs (or other detailed logs)
• SQL Server Profiler

However, system administrators generally think of these tools as troubleshooting tools and not monitoring tools. The difference is that monitoring is run regularly and is a process for discovering problems. Troubleshooting tools on the other hand are manually run by the administrator as a reaction to a problem.

In order to start having a ground level view, these tools need to be deployed for monitoring purposes, not just troubleshooting. In order to do this these tools should be scheduled to collect data for a period each day. They should all run at the same time so you can correlate the data. Then the data needs to be analyzed, and the relationships between different sources on a regular basis.

The Attributes of Micro Data

The most distinct attribute of ground level data is that there is a lot of it. This attribute has several consequences for this type of monitoring:

• Different sets of tools to process the data are needed
• Generally samples of high resolution data and not complete sets are used
• To correlate all of this, you will need to do some work because it will be different for every environment

Since you are working with samples to make the data size manageable it is good to think about what your samples represent. They might not show things that happen say every hour if you have a 20 minute sample. If you are choosing a set or single server from a farm, you might miss issues that are particular to one server. However, mid-level monitoring like Nagios are usually good for finding these problems. Also, if you choose your samples well you can likely discover things your standard monitoring systems miss.

One other thing to keep in mind is that collecting high resolution data can be resource intensive, so it is possible that the act of monitoring effects the system itself (And no, I am not going to cite a certain physics principle).

Case Study: Web Logs and Perfmon Data

Data Analysis Platforms

Having a platform to work with for data analysis is essential for high resolution data. I have been learning R and using RStudio as my data analysis platform for a week now. I believe this is going to be my standard tool for analysis. R is a domain specific language focused on statistical analysis. Platforms such pysci and R are going to become part of the standard toolkit for system administrators because they allow you to view your data in different ways (i.e. distributions) and provide a lot of functionality to combine different data sources. They are also naturally more programmatic then something like excel.

Getting the Data

For windows the standard tool to get a high resolution picture of the system is Perfmon. With “Data Collector Sets” you can give a list of counters to monitor and save to a perfmon binary file (.blg) or other formats. I used this to collect data from 20 minutes on one of my web servers. We also insert our web logs into SQL Server. Both of these allow me to easily extract CSV files which can be imported into R. Since the web logs are in SQL, it is easy to filter on time and requests that only went through the web server I was monitoring with perfmon.

Exploring the Data with R

I have a feeling as I get to know R better I will discover more advanced ways to mine the data for things I am interested in. For now, plotting things like CPU and response time of web requests lets me visually find correlations without too much difficulty. After starting with CPU and web response time, I noticed that there was a correlation. Digging into this more, I found that response time for several requests will go up to over a second when the CPU spikes to around 60%. This also correlates with .NET lock contention as well (Larger Image, R Script):

Our SQL logging is currently losing entries, but there is enough data to see the correlation. I suspect with a full weblog data set the correlation will be even stronger. I have yet to find out the cause of these slowdowns that seem to happen about every minute, and am going to enlist some developers to get to the bottom of it (“Correlation does not imply causation”). I should point out that I checked another sample of weblog data when perfmon was not running to make sure the response time spikes were not a result of the monitoring itself.

The Difference?

If we looked at the CPU from something like Nagios it is going to look quite low (20%). This problem is unlikely to show up when using a profiler unless you just happen to load the page during these slow downs. High resolution monitoring allows you to discover issues like this that get buried with low sample rates. Also having a platform to view multiple data sources allows for the discovery of correlated metrics.

What is Next?

I don’t have a high resolution system deployed yet. So I need work on and think about:

• Scheduling Perfmon
• How to get the most representative samples
• Scheduling TCPDump and getting that data into R
• Scheduling Sar on the Linux boxes and getting that into R
• Automating parts of the analysis

I also need to discover more efficient ways to discover correlations and patterns in R with the data I am collecting. In short, the need for high resolution monitoring is becoming evident to me and there is a decent amount of work to get this deployed. As I explore this domain of monitoring I will get to know the caveats and develop systems for doing it more effectively.

In the next post I will talk a little about “Day to Day” monitoring with systems like Nagios, and how this data compares to higher resolution data in attributes and functionality.

Transfer rates and the number of packets you send are measured in units of a certain quantity of data per units of time. The unit of time that everyone is used to is the second. The standard quantity of data that is used in the networking field is bits and the standard time unit is seconds. So for example, the standard network interface these days is 1 Gigabit per second. So the quantity of data is a Gigabit, and the unit of time is a second. We call this the transfer rate. The key thing to remember is that this is a fixed ratio of data over time. Because of this, you can divide the ratio by any number you want to (Ignoring the complexities of the discrete properties of Ethernet frequencies, system clocking, etc). So, 500 Mbit over a half second is the same fixed ratio as 1 Gigabit per second.

The thing is though, in computing, a second is a really, really, really long time. This is important, because when we choose what unit of time to express this in, what we are doing is graph smoothing (It is sort of, although not really, like taking an average).

For example, we could transfer 900 Mbit in half of a second and another 100 Mbit for the other half of that second. How much data was transferred during that second? The answer is 1 Gbit. If we transfer 500 Mbit per half second and another 500 Mbit per the other half second — this is also 1 Gbit per second:. This effect is illustrated in these Megabits per half second graphs:

These two are clearly not the same thing, but when you express them as the amount of data transfered over a second they are. This is important because a 1 Gbit per second interface is also a 500 Mbit per half second interface — and a 500 Mbit per half second interface can’t transfer 900 Mbits per half second (I am ignoring any buffering effects, but in practice we have found this to be essentially true).

This effect is made even worse by most monitoring tools because most take samples every 5 minutes. So what you are really seeing is the transfer rate per 5 minutes converted to a per second rate. This sort of thing is why people say data can lie.

Why Should you Care?

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

We have bonded these interfaces using Intel Load Balancing (ALB/RLB) and for the most part our discards have gone away. We did this on all but one of our web servers for a while and found that the one that didn’t have the bonded interface had discards climbing while the others did not.

A second is a long time — be wary of trusting it too much to measure things.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

• When the test was run in UTC time and who ran it
• Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
• Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
• Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.

With all my talk of doing things in a scalable way comes a requirement, and that is that we actually accomplish this stuff in practice at Stack Exchange. We have been making a lot of progress in this area. George has been refining and expanding our deployment process. He improved our Windows deployment to include most of the software we use and has made it so kickstart for CentOS/Linux installs are integrated into our deployment as well.

Scaling your ability to manage your environment in my mind means doing more with less. I find I really only have to ask myself one question to quickly gauge if we are doing it right or not.

Do I have to repeat steps to do this task on multiple servers?

I like this question because it is more specific than “Is it automated?” while still implying automation. It is akin to the DRY rule in programming: “Don’t repeat yourself.”

When it comes to our environment, here is where we are at. For this I will ignore some details — for instance, we log into servers to kick off a PXE install and then just let it go — I consider that a Yes to “No repetition required” since it is only one or two steps and we don’t really deploy more than a server at a time. We don’t want to automate to the point over engineering beyond the projected size of our environment.

¹We do have some software that can be deployed via GPO, and LiteTouch deploys a lot of stack on the web servers during deployment. But future software updates and software that doesn’t lend itself to GPO is manual.

† I am currently in the middle of rolling out puppet so it is partially deployed on some of our Linux servers

The big picture of all of this is deployment as phase 1 and maintenance as phase 2 for both Windows and Linux. Also, ideally these phases connect to each other seamlessly.

One of our main goals is to change all of the above “No” to “Yes” over the next few months and then refine each step. For the most point we have deployment taken care of for both Windows and Linux. As far as maintenance goes, I believe as I progress in rolling out Puppet that will solve the vast majority of our Linux needs. How we will manage our firmware is still an unknown. As far as Window maintenance goes we are going to start exploring System Center soon and hope that it can meet our needs.

What I really think all of this will buy us is consistency, recoverability, and most importantly — time. By having all of this centrally managed it will make our servers more consitent with each other — and make them effectively drones. By having these processes automated we will be able to recover fast and replace servers easily. Lastly, and most importantly it buys time. By making our management faster and more agile, George and I can focus on rapidly deploying improvements to our environment instead of just maintaining it. By having less friction to deploying changes to our infrastructure I believe more possibilities for improvement will start emerge.

Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.

The League of Professional System Administrators, LOPSA, is a non-profit organization aimed at advancing the profession of system administration. Like Server Fault, they want to see the knowledge and skill of system administrators expand. To this end, they do several important things which include local meetups and regional conferences. Also, both Server Fault and LOPSA are vendor neutral, we don’t care if you are a Cisco Network administrator or Linux administrator, we both want to see the field of system administration united.

So, why is this important to you? Server Fault has partnered with LOPSA to give 40 of our top users free memberships to LOPSA.

Both George and I have started to attend our local monthly meetings in New York (LOPSA-NYC) and Boston (Back Bay LISA). I asked George about how he felt about his LOPSA-NYC Meetings:

LOPSA-NYC, even though it is in it’s infancy, has been a great place for learning about methods and technologies I don’t get to interact with on a daily basis. That alone is a great thing, since – in my opinion – the goal of a sysadmin should be to learn about more than just the narrow set of technologies presented to them in their day job. A second, and some including me might argue a more important benefit out of going to the lopsa-nyc meetings is that I get to meet and talk to real people (err sysadmins, but close enough). The time after the talks is really where the advantage of LOPSA shines through, you get to talk to people that have seen the same issues as you, people who may have a new insight in a problem that you have been having that shines the light onto the solution.

LOPSA also puts on local conferences, out in Seattle they put on The Cascadia IT Conference. They also recently had PICC in New Jersey where both George and I gave a talk about Stack Exchange’s architecture.

Whether you call it DevOps, system administration, or system engineering our field is growing as a profession every day. Through the knowledge shared at conferences and local meetings like Cascadia and PICC, LOPSA’s mentor program, and Server Fault we constantly advancing, and these memberships are meant to be a part of that. We hope you are interested, and can take advantage of local chapters or decide to start your own. See this meta.serverfault.com question to put your name in for a membership — at the end of a week or two we will sort by reputation earned this calendar year, take the top 40, and pass your email address on to LOPSA for your membership.

Look at your infrastructure and the way you handle your operations and ask yourself:

“Are we more like the Borg or the Federation?”

If you are not sure, one way you can tell is to look at the pieces of your infrastructure and see if they have personality. If a server has personality, then it is more like the Enterprise. There may be sister ships, like the USS Yamato, but it has its own personality because Geordi has made improvements and its crew has made it unique. Disgusting.

Why disgusting? Personality is the antithesis of scalability. With a large number of servers, each with their own personality, you will need more administrators, and a chaotic management structure. The Borg are the good guys. This is because of some of the distinct advantages of being more like the Borg:

• In the Borg collective, individual drones and ships can be destroyed without much thought. They are easily replaced.
• There are different types of Borg vessels, cubes, spheres, tactical cubes, but within these types, there is no personality.
• Improvements that are assimilated become part of their whole collective making them highly adaptable

In IT operations, your servers should be drones that are easily replaced. You may have different classes of servers, but if they are all the same within each class they are easier to manage. For example, if you have a different brands of hard drives and RAID cards in each server, you will have spend time figuring out the settings, tweaks and compatibility issues for each of these different types. You will also have to track updates for each of these two different things and figure out methods of deployment for each of these types if a vendor doesn’t provide tools to manage these. With configuration and centralized management you can update and adapt improvements to all your servers rapidly. Deploying a change to all servers is just a script or a policy, not a manual process for each server. If you hand code each configuration file and manually deploy software they will start to become different over time when mistakes are made — this is one of the main ways servers develop personality.

At Stack Exchange, we have done a good job at achieving this with our web tier. We do have 1 staging web server, but for our other 9 web servers, the seventh is just seven of nine. George has made a deployment process which includes everything we need, making any web server disposible, and a new one ready for assimilation. Should one of them be destroyed, the others will automatically take over without concern. However, in some areas we are still more like the Federation. We have 4 database servers, and we have noticed areas where our db01/db02 pair is unlike our db03/db04 pair. If one of our primary servers fail, we will mourn the loss as we carry out a manual fail over process. If we were more like the Borg in this area, we could initiate self destruct on one of these servers without a care as the Borg queen would do.

The Borg are the role model, not the Federation. Next time you look at a server that has personality, your first thought should be: “You will be assimilated.”

At a recent talk at PICC, part of my introduction was:

“Hello and thank you for coming to our talk. This is George Beech and I am Kyle Brandt, and we are the two sysadmins at Stack Exchange…”

While “two” is the technically correct answer going by our self-anointed titles, it isn’t really the whole picture.

The Sysadmin Continuum

In reality here at Stack Exchange we all wear a lot of hats. While George and I may own the system administration tasks, we work together with our developers who are pretty accomplished sysadmins in their own right. If it were just George and I soely doing the work that could be considered system administrator work, we would probably have a pretty tough time of it at only two people.

In reality we are continuum. System administration is just a spot on the continuum where George and I spend most of our time. However, a lot of our developers spend a lot of time there and travel the same road.

Some recent examples:

• Jarrod Dixon worked with me on developing a log daemon for inserting HAProxy syslog data into SQL Server
• Nick Craver frequently hops on Dell and makes hardware recommendations
• Jeff Atwood helps us with SSD research
• Sam Saffron works on SQL restore scripts
• Geoff Dalgas racks servers out in OR and troubleshoots our NAS
• Ben Dumke and Jeff Atwood do some CDN research
• Jeff Szczepanski, our Vice President of Products, analyzes packet dumps

Because of the spirit of shared responsibility, we get to take advantage of everyone’s knowledge and experience and we are all better for it.

So in the end, “How Many System Administrators does Stack Exchange Have?” is a question I can’t really answer.

A couple of weeks ago we had one of our edge routers go down on us. Nothing bad happened, failover to our secondary router work just as expected. Now, we saw something wierd this week when we looked at the internal interface graph for our secondary router.

I’ll give you a second to try and see what we saw – although I don’t think you’ll need a whole second to see something very very strange going on with this router. That’s right, there is a whole lot of outbound traffic on this router, but ZERO inbound traffic. The next question we had was what could possibly be causing this? I really don’t think that I could be anything good.

You can clearly see on the graph where we failed back to our primary router. But, after that there is still a ton of traffic that is traversing our secondary router when there should be very little traffic going through there.

After some digging around we found that our Windows servers had the wrong ARP address for the VIP of our routers. That’s right, Windows still had the wrong ARP address after the fail back to the primary router, it even had the wrong ARP address days later.

How could this be possible? I was stumped after some digging around it seems that Microsoft changed the way that the network stack handles Gratuitous ARP packets (GARP packets) with Windows Vista/2008 RTM. This change has persisted through to Windows 7 and Windows 2008 R2.

What is Gratuitous ARP?

Gratuitous ARP is when a system sends out an ARP packets announcing to all system what it’s MAC address is. You generally see these in HA environments that make use of Virtual IPs that can move back and forth between machines. You will normally see a machine issue a GARP packet when a fail over event occurs and the new machine picks up the VIP. Wikipedia ARP article for more info on the ARP protocol

What did Microsoft Change?

There is actually very little information out there about windows and how it handles GARP. The best resource i’ve found that gives a very good overview of the new windows networking stack is a very well written technet blog. About 3/4 of the way down there is a section named “Changes to ARP cache updating” within this section lies the answer to all the mystery of our weird network bandwidth.

>First, a Windows Vista or Windows Server 2008 will not update the Neighbor cache if an ARP broadcast is received unless it is part of a broadcast ARP request for the receiver. What this means is that when a gratuitous ARP is sent on a network with Windows Vista and Widows Server 2008, these systems will not update their cache with incorrect information if there is an IP address conflict.

>Additionally, when a gratuitous ARP is sent by a Windows Vista or Windows Server 2008, the following change has been made – the SPA field in the initial request is set to 0.0.0.0. This way the ARP or neighbor caches of systems receiving this request are not updated. So, if there is a duplicate IP address, the receivers do not need to have their cache corrected.

The question is why is this such a big problem for Microsoft? Well the answer is they have hi-jacked the GARP packets for their Address Conflict Detection mechanism. You know that pop-up that says “Another machine on this network has been detected with the same IP address”. With previous version of windows they had the same mechanism, but still respected the normal GARP packets, thus there would sometimes be an issue with Windows systems updating their ARP cache with invalid data. They fixed it by breaking GARP.

Why is this a problem?

Beyond the issue we have seen with Windows 2008 not respecting GARP packets this can cause other wierd problems. One example I can think of off the top of my head, is that for HA systems that use GARP to facilitate moving the VIP when a system goes down is that you will now have to wait for the OS to timeout the neighbor cache. This will add more time to your fail over, possibly causing things that are expecting a quicker fail over to break.

Is there a Fix?

I have not been able to find a fix for this. Although there is very little information out there on Windows networking at that low of a level. If you know of a fix to this issue I’ve started a question on Server Fault.

In the life of a company and in the life of a person there is a line that gets crossed when it comes to being organized. When you start out in a business basic documentation and good organizational systems are often optional.

To be clear, I am not saying that they should be optional, rather, that from my experience this is just the reality of the situation. There is usually only a couple of people working on something, so they can always ask each other things and the amount information is small enough that people can mostly keep it in their heads. Also, to be more clear, I don’t really think this is the best way to do things, by its very nature it is inefficient. But in a fast moving startup, it is easy to give into the temptation to skip some of the niceties and get to the next project.

But then one day, you start to get close to the Organizational Event Horizon or OEH. The way you will realize it is by the way it feels, it is the feeling of being Spaghettified. In other words, you get stretched beyond what you can handle. Before you cross the OEH, good organization is about being more efficient and getting more things done. But once you have crossed the OEH, organization becomes the difference between success and failure. Important and critical tasks will start to get missed, and the stress of all the different things going on will tear you apart. If you don’t start to deal with the situation before crossing the OEH, failure in inevitable.

How To Handle It?

I feel like at least on the sysadmin things here at Stack Exchange I am starting to feel the pull. To fix this I think the key is to implement a few basic systems for organization that are based on two related principles: The KISS (Keep it simple stupid) and the “stub” principle. The idea behind both of these is to have the minimal amount of resistance to actually start using documentation and organizational systems. The KISS principle is well known. What I call the “stub principle” is something I picked up from wikipedia and Clay Shirky’s Here Comes Everybody:

“…as long as the experts did nothing (which, on Nupedia, is mostly what they did), nothing happened. In an expert-driven system, an article on asphalt that read “Asphalt is a material used for road coverings” would never appear, even as a stub. So short! So uninformative! Why, anyone could have written that! Which, of course, is one of the principal advantages of Wikipedia. In a system where anyone is free to get something started, however badly, a short, uninformative article can be the anchor for the good article that will eventually appear. Its very inadequacy motivates people to improve it”

In short, something is better than nothing, and there is never really an excuse not to at least start a “stub.” Then if someone is unhappy with the stub at some point, they can improve it. The hardest step for many people, actually starting something, is already done.

In system administration, stealing from Tom Limoncelli, I think the best “stub” is usually a checklist for most system administration stuff. So for example I just started one for “Deployment Steps” which doesn’t actually include how to do anything — just what to do. If someone wants to, they can easily add to it later. Other methods include spreadsheets in Google Docs for licenses and IP address lists, and very short “meeting minutes” which just include decisions made at the meeting and things people said they would do.

On the personal side of things, I have started to use the Inbox Zero technique and Remember the Milk for my checklists. Both of these help me keep my head clear and “Inbox Zero” helps me make sure I don’t miss emails. I have also started to try to batch my interruptions by checking email and chat every 20-30 minutes instead of constantly for at least part of the day.

It doesn’t matter if these are the ultimate-super-top-of-line-over-engineered organizational systems, what matters is that they are simple enough to use and get started with so we don’t pass the point of no return.

(P.S. Before you point out all the obvious flaws in my analogy and that I’m mutilating popular physics, please read Miguel de Icaza’s Well, Actually)