Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.

We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

• Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
• Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
• Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
• Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

• There are constraints of what is actually available to buy, and how much you can actually know what you are getting
• Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.

When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

• Gave us time to rectify some mistakes
• Gave us confidence in our secondary datacenter since we failed over to it before the storm
• Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.

Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

StackExchange welcomes our newest system administrator, Steven Murawski!

Steven followed a bit of an eclectic path to systems administration, having started out operating a garden center, floral design and landscape business. After eight years he decided to move into law enforcement; he went to school for Police Science and worked at a local police department as a clerk, dispatcher and auxiliary officer. While he enjoyed shooting various firearms and driving at high rates of speed, he also began working with the police department’s IT systems, and slowly fell in love with systems administration. He became a member of LOPSA, Microsoft MVP, and continues to lead two local user groups (the Greater Milwaukee IT Pro User Community and Greater Milwaukee Script Club). Steven combined his love of systems administration with public speaking by presenting technology talks at various technology conferences and user groups.

Today Steven distracts himself from technology by playing with his eight month old son, puttering around with home-improvement projects and hurling pieces of lead through paper targets, although he is seriously considering once again picking up the mantle of host for a systems administration and Powershell podcast!

I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

• Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
• Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
• Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
• Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

Like any good family, when we saw that our older sibling had something we wanted what did we do? Well, we whined and complained until we got it!

Stack Overflow recently finished their 2011 survey, and as soon as the results where announced, Server Fault being the proper younger sibling that it is wanted one too.

So I call all of you Server Fault users to come and take our survey, you wanted it and now you’ve got it! (oh and you really should get your sysadmin friends to take it too – the more data the better!)

Just like the Stack Overflow survey we will be releasing the data right when the survey ends!

At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

• Monitoring allows you to send alerts if certain conditions are met,
• It allows you to visualize trends in data,
• Provides a method of assurance to the customer that their consumed services are guarded,
• Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.

One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

We have gone and done it, we’ve expanded our Systems Administration team to THREE people. I’d like to introduce you to Peter Grace who we have just brought on board as our third sysadmin.

This is the second company I’ve had the honor of working with Pete at, and I can say that he is an exemplary System Administrator. In addition to being a great sysadmin, Pete is a fellow gun enthusiast, which increases our zombie attack survival rate by 35%.

Pete and I have done many great things together in the past, and I’m very excited about what he brings to us and the great things he, Kyle and myself are going to do as we move forward doing our part to make the internet better.

Recently there was a question on the Programmers site asking “Why don’t all companies buy developers the best hardware?” This is actually a very interesting question and there was a good deal of discourse on the topic, as there always is when you get a group of professionals talking about the tools of their trade.

We here at Stack Exchange pride ourselves on not letting the technology get in the way of our devs. We want them to be able to do their job – which is writing code, and doing it well – with minimal hassle. Now, I have to take a brief moment to make a little bit of a note some of the things we do do not scale.

The philosophy that is maintained when getting a developer a new machine is a pretty simple one:

> If a dev is constantly struggling with their machine they aren’t getting work done, they aren’t happy, and they are producing bad code. All of these things are more expensive than a nice developer machine

As a sysadmin – and hence the guy in charge of getting the devs what they need you find out really quickly that they know what they want for the most part, and for those that you don’t ask if you give them a powerful machine with a lot of screen real estate they are generally very happy.

Since I mentioned earlier that there are some things we do here that aren’t all that scalable – in that they work for a company with 20 devs but not 100 i’ll split them up.

The Scalable

The base config that I get new devs in the NY office is:

• Dell Optiplex 980 class tower
• Max out the RAM
• Best i7 Processor that I can get in it
• SSD primary drive
• 7.2k large (500GB – 1TB) Secondary Drive
• 30″ Primary Monitor
• 20″ Secondary monitor (turned sideways)

Every dev picks their own keyboard – we may give them the crappy one that comes with the machine to get them up and running, but they can request any keyboard/mouse combo they want (and … i do mean any).

The not so scalable

• For our remote devs we get workstation replacement class laptops. Basically I go out and find the most powerful laptop I can get at the time they start, as well as a 30″ monitor to go with it. Not Scalable.
• At least one dev in the NY office has a fully customized hand built machine with 2 30″ monitors. VERY not scalable.
• Whatever they want – within reason

Basically, it boils down to get them what they need to get their job done. One of the biggest challenge to some people is that they do not have management’s buy-in to get the Devs what they need. I find that this is a sad state of affairs, but I have no real advice to offer – especially since I work somewhere that management has mandated great dev machines.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

• When the test was run in UTC time and who ran it
• Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
• Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
• Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.

With all my talk of doing things in a scalable way comes a requirement, and that is that we actually accomplish this stuff in practice at Stack Exchange. We have been making a lot of progress in this area. George has been refining and expanding our deployment process. He improved our Windows deployment to include most of the software we use and has made it so kickstart for CentOS/Linux installs are integrated into our deployment as well.

Scaling your ability to manage your environment in my mind means doing more with less. I find I really only have to ask myself one question to quickly gauge if we are doing it right or not.

Do I have to repeat steps to do this task on multiple servers?

I like this question because it is more specific than “Is it automated?” while still implying automation. It is akin to the DRY rule in programming: “Don’t repeat yourself.”

When it comes to our environment, here is where we are at. For this I will ignore some details — for instance, we log into servers to kick off a PXE install and then just let it go — I consider that a Yes to “No repetition required” since it is only one or two steps and we don’t really deploy more than a server at a time. We don’t want to automate to the point over engineering beyond the projected size of our environment.

¹We do have some software that can be deployed via GPO, and LiteTouch deploys a lot of stack on the web servers during deployment. But future software updates and software that doesn’t lend itself to GPO is manual.

† I am currently in the middle of rolling out puppet so it is partially deployed on some of our Linux servers

The big picture of all of this is deployment as phase 1 and maintenance as phase 2 for both Windows and Linux. Also, ideally these phases connect to each other seamlessly.

One of our main goals is to change all of the above “No” to “Yes” over the next few months and then refine each step. For the most point we have deployment taken care of for both Windows and Linux. As far as maintenance goes, I believe as I progress in rolling out Puppet that will solve the vast majority of our Linux needs. How we will manage our firmware is still an unknown. As far as Window maintenance goes we are going to start exploring System Center soon and hope that it can meet our needs.

What I really think all of this will buy us is consistency, recoverability, and most importantly — time. By having all of this centrally managed it will make our servers more consitent with each other — and make them effectively drones. By having these processes automated we will be able to recover fast and replace servers easily. Lastly, and most importantly it buys time. By making our management faster and more agile, George and I can focus on rapidly deploying improvements to our environment instead of just maintaining it. By having less friction to deploying changes to our infrastructure I believe more possibilities for improvement will start emerge.

Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.