Our New York Peer 1 datacenter at 75 Broad is still running on generator power, but as a precaution we decided to failover Stack Overflow, Careers, and the rest of the Stack Exchange network to our secondary datacenter in Oregon last evening. It turned out to be the right call because the refueling trucks can’t get to the facility, so Peer 1 is shutting down all power in about 30 minutes.

We actually recently tested a lot of this, but this is our first time failing over everything at once. So far it is going pretty well, but we have run into a few issues so far:

• An index reorg job kicked off right before failing over. This meant that our SQL replication partners across the country were 40 Gigabytes behind. So Stack Overflow had to remain in read only for about an hour
• Because the status message on our sites is stored in the database, that was readonly, so we couldn’t update it to let everyone know it would be read only for about an hour
• We realized we have to transfer the AD FSMO role forcefully since the NY DCs were shut down, and we don’t know how much fuel is left
• Our backup monitoring system isn’t permitted as an SNMP manager via the Group Policy, so we have to update that

We have some open concerns, and will be keeping a close eye on the following:

• Oregon has some lower end Dell switches, we hope they handle the load. We will be shipping the current 2960S switches to OR once we upgrade our NY switches to the Nexus 5k/2k line in a couple of weeks
• Our load balancers out in OR are a little tight on CPU
• We have 5 web servers in OR instead of 10. However, the combined CPU load of NY on the web tier is ususally 100-200% (Out of 1000%) so I think it will be okay:

However, in the big picture, we have successfully failed over to Oregon! Today is going to feel like taking the sub below its depth rating, you can watch the Das Boot Video to share our feelings.

Before our test last weekend, we posted THE PLAN, as promised here’s a follow-up of how things went:

• Prep (2 hours before the test)

1. Shorten DNS TTL down to 5 minutes
2. Pause page duty (that’s damn sure going to go off)
3. Firewall Oregon redis to prevent mutation (went smooth, late plan addition)
4. Slave Oregon redis from the New York master (smooth, late addition)

• The Test

1. Shutdown affected backends in HAProxy (New York)
2. Start the DNS swap to Oregon IPs
3. Start the SQL 2012 Availability Group failovers to Oregon (largest problem)
4. Drop redis firewalls in Oregon (went smooth, late plan addition)

• Wait for this to complete before moving forward

1. Sanity check sites on the Oregon web tier
2. Enable the backends in HAProxy (Oregon)
3. Bring the sites out of read-only mode (can be improved)
4. Find problems, squash bugs in our configuration until we’re running smooth (went well)
5. Firewall New York redis to prevent mutation (broke OpenID)
6. Slave New York redis from the Oregon master (smooth, late addition)
7. Slave New York redis backup from the New York master/slave (smooth, late addition)
8. Oregon went totally offline, twice!

• Failing back to New York

1. Shut down backends in HAProxy (Oregon)
2. Start the DNS swap to New York IPs
3. Start the SQL 2012 Availability Group failovers to New York
4. Drop redis firewalls in New York (smooth, late addition)

• Wait for this to complete before moving forward

1. Sanity check sites on the New York web tier
2. Enable the backends in HAProxy (New York)
3. Bring the sites out of read-only mode (wasn’t actually needed)
4. Get beer (check, check)

Some of these were late additions to the plan.  Having redis be a warm cache once we were up in Oregon meant a few more steps added to the original plan, but well worth it.  A cold cache for all sites means stumbling of the servers and slow page loads for the first wave of hits…why have slow pages when they can be fast?  The above is a high level plan…the actual one has even more small steps in there, so let’s look at what failed at a high level and some of the smaller details as well.

Fails

• Time-wise, the biggest issue was the SQL 2012 always on availability group failover for our SENetwork_AG group; this group contains all of the databases for sites that aren’t stackoverflow.com.  While the StackOverflow availability group failed over across the country in seconds, the much larger SENetwork_AG (by database count – that’s what matters in our case) did not.  Here’s how that one played out:
  • (+0 min): Failover of the SENetwork_AG begins
  • (+5 min): After attempting to failover via the SSMS GUI and saw a timeout after 5 minutes
  • (+6 min): We attempted to fail it over via script in case this was a tooling timeout in plan
  • (+11 min): It’s not a tooling timeout; time to up the default timeouts on the listeners and AG resources in windows
  • (+16 min): This had no effect, the 5 minute timeout is somewhere else in the pipe
  • (+17 min): As a last ditch effort to get the AG ownership moved to Oregon, I disabled the AG’s dependency on the listener (which we don’t want or need, but have to have)
  • (+17.5 min): Success, AG is spinning up
  • (+19 min): All databases are back online, SE 2.0 sites are now up
  We don’t fully understand why this works in New York but not in Oregon…our understanding is it should fail or work equally in both cases, but obviously that’s not correct.  I’ll be following up with Microsoft on what we learned here in hopes it can be improved.
• The second most visible failure was Oregon going completely offline, twice!  We have traffic, lots of traffic.  This means lots of simultaneous connections to our load balancers, especially when we’re coming up from an outage.  This means the default conntrack limits in CentOS 6.3 on our HAProxy load balancers weren’t high enough.  We solved this by upping the limit to 1,048,576, matching New York (it turns out we did this weeks ago…fail #2 revealed why it didn’t stick). Later, after another puppet deploy (we have things templated to keep 2 datacenter in sync so…puppet!), the iptables service reloaded.  This caused CentOS to unload/reload the iptables module resetting the limit…causing another outage, hoorah.  We fixed the limit again and then prevented further reloads – problem solved.   This was a good pair of lessons we can apply for when New York load balancers are fully under puppet control.
• The third, lesser-noticed failure was that when we began the redis slaving back to New York to keep that warm cache, we blocked another service using that redis instance: Stack Exchange OpenID.  Once we identified this issue we moved it to another instance that isn’t slaved or firewalled as part of a failover.  There would be a similar problem when we test OpenID, Careers, etc. failover in a few weeks…so this fix takes care of things for that test as well.

Things that can be better

• When the sites were available (open via HAProxy) but the databases were not yet online, we broadcast a raw error page (YSOD) to users.
  • While this can be fixed by not opening the HAProxy backends until the sites are ready, we prefer to at least know what was throwing that error.
• Bringing sites out of read-only mode was more tedious than anticipated
  • We have a “disable read-only” button per-site…I’ll be adding a global one as well for situations like this
• Exceptions logging needs some thinking. Our exceptions log to a database that was failed over to Oregon, making it read-only in New York.  This meant the services that didn’t failover in New York trying to write to that database had to queue up their exceptions and write them out to the database when it was available for writes again.
  • While this was an excellent test of StackExchange.Exceptional’s error queuing in case of database failure…we’d still like better farm-wide visibility during a partial failover.

Overall, we were very happy with how this test went.  Most issues were identified and solved quickly, and most of our fears were laid to rest.  This has been a long, hard effort by many devs and sysadmins on multiple teams…and we’re not close to being done.  This test going very well for the most part has been a very rewarding payoff on our side, we’ll keep you updated as our datacenter move progresses.

This coming Saturday, October 13^th around lunch 3PM UTC we’ll be testing our redundant datacenter and failover procedure.  We’re hoping for two brief downtimes while we swap over and swap back between New York and Oregon are all the outside world is aware of.  We’re planning for mass chaos and lots of fail.

Over the last few months we’ve been beefing up Stack Overflow’s original home at PEAK internet’s datacenter in Corvallis, Oregon.  Here is the list of shiny hardware now packed in Oregon: 4 new Dell R610s (Web), 2 new R620s (1 Web, 1 Redis), 2 new R720xds (1 DB, 1 Logs), 1 new R710 (DB), 1 recycled R710 (services), 2 recycled R610s (routerbalancers). I’ll detail what the setup is in another post.

So here’s what we want to happen this weekend:

• Prep (2 hours before the test)

1. Shorten DNS TTL down to 5 minutes
2. Pause page duty (that’s damn sure going to go off)

• The Test

1. Shutdown affected backends in HAProxy (New York)
2. Start the DNS swap to Oregon IPs
3. Start the SQL 2012 Availability Group failovers to Oregon

• Wait for this to complete before moving forward

1. Sanity check sites on the Oregon web tier
2. Enable the backends in HAProxy (Oregon)
3. Bring the sites out of read-only mode
4. Find problems, squash bugs in our configuration until we’re running smooth

• Failing back to New York

1. Shut down backends in HAProxy (Oregon)
2. Start the DNS swap to New York IPs
3. Start the SQL 2012 Availability Group failovers to New York

• Wait for this to complete before moving forward

1. Sanity check sites on the New York web tier
2. Enable the backends in HAProxy (New York)
3. Bring the sites out of read-only mode
4. Get beer

What will happen? We don’t know – but we’ll blog here about it.  The whole team has been working on various pieces needed for failover and we’ve tested as best we can (and we will continue to test the rest of this week).  The reality is that we can only test so much and some things don’t break until you release production level traffic on them, that’s what this weekend is all about.

All of this is leading up to a datacenter move in New York where all our servers will be taking a trip a few miles north where we’ll have a bit more room to grow – we’re building out that datacenter now for a move in about a month, we’ll try to have lots of info and pics on that build out.

Here are a few things we’re afraid (again, why we’re testing):

• Internal API wonkiness while we’re switching DNS (our sites talk to each other)
• Bad DNS caching on misconfigured servers/proxies
• SQL 2012 Availability group failover may not work (so far as we know, no one’s tried one with 200+ databases, much less to another datacenter with that many)
• 5 web servers can’t handle the load (services are overlapping more, potentially causing more memory utilization than in the 11 server farm in New York)
• Oregon doesn’t have enough upstream bandwidth

Besides being a downtime notice, this post will serve as a basis for comparison for when we blog about which parts of the above plan went horribly, horribly wrong.

It is pretty hard to get a bunch of system administrators together for any period of time and not have a conversation about command line for managing things. In my experience, command line always wins the debate for any medium to large installation. Microsoft caught on to this and has introduced powershell. From my own experience and people I have talked to, powershell adoption has been slow. Like powershell or dislike it, it is an interesting take on command line with its object oriented versus text-based approach. It is also quite powerful, so why the slow adoption?

In my mind it isn’t a problem with powershell itself, but rather a result of human nature, culture, and the Windows ecosystem. The problem lies in workflow, and I think the best way to illustrate this is to contrast Windows versus Linux administration. When managing Linux systems, scripting something flows naturally from the experience of trying things out in the first place. For example, lets say someone with some Linux experience (but not an expert) is building a piece of software and installing it on Linux for the very first time. The steps usually involve something like:

1. Run the traditional, configure, make, make install from the command line. See what libraries or binaries are missing that are required to build, from the errors. Install them (often via yum or apt), keep doing this until it works.
2. Possibly add some users to the system, adjust permissions etc with commands like useradd, chown, chmod.
3. Edit some text files
4. Set it to start at boot with commands like chkconfig or update-rc.d

As you do this more and more, you learn the patterns of what fails, and get the urge to automate it. Although things like puppet are the end goal these days, the next normal step is to write shell scripts to do this. This is the lynchpin of the whole experience, a shell script is just a little flow control, error handling, and a few variables on top of what you were already doing. With the Linux experience, scripting flows naturally as a next step in your experience. It is a gradual learning experience, and your previous experience of doing things manually taught you much of what you need to know to script it — what goes wrong, what to check, how to do everything you need to do. Even the first time tutorials on the web are almost always command line. From the second you start learning how to administrate a Linux box, you are learning how to script.

Unless you are extremely disciplined or have been taught Windows administration recently by a “benevolent” dictator, your first steps in Windows administration are via the GUI and Wizards. Don’t get me wrong, for many things this gets the job done faster and with far more ease. The generally accepted problem with the GUI though is that it doesn’t scale. When you want to scale in management you often turn to scripts (GPOs are very useful, but you are generally out of luck if what you want isn’t predefined.) With Windows, scripting these tasks is generally a complete departure from your previous experience. Scripting doesn’t flow naturally from previous experience, and scripting something is scary because of this. This means the windows ecosystem, due to human nature, is far less likely to produce administrators with scripting experience. Therefore, powershell adoption is logically slower. This also means administrators are less likely to be testing powershell and pressuring each other and Microsoft to improve it.

The cure to this workflow to me seems to be Windows Core since changing human nature isn’t easy. With Windows core, even though you can use the Management Console, you really have to start doing many things via the command line and powershell. With use of these tools, just as with the shell in Linux, scripting will flow naturally. The Catch-22 however though is that since adoption is slow, everything you need might not be available via command line options. At my company we haven’t used Windows core, but I’m hoping with Server 2012 it is time for Core, does anyone with more experience know if it is time for this shift?

Sorry this blog has been a bit quiet lately, we’ve been very busy making some big changes behind the scenes.  So what are we up to?  Let’s start with just the SQL infrastructure moves, here’s a list of servers in play as they started out:

• NY-DB01 – SQL2008 R2 Hosts all sites except Stack Overflow and the Sites DB
• NY-DB02 – SQL2008 R2 Daily backups restored from NY-DB01, and the Dev DBs
• NY-DB03 – SQL2008 R2 StackOverflow’s DB
• NY-DB04 – SQL 2008 R2 StackOverflow 5 minute behind hot spare in restore mode
• OR-DB01 – SQL2008 R2 Chat’s DBs
• OR-DB02 – SQL2008 R2 SEDE, Internal SEDE and Chat Dev DBs
• New Dell R710 w/ 2x OS Drives + 6x Data 300GB Intel 320 SSDs in RAID10 and 96GB RAM
• 3xNew Dell R720 w/ 2x OS Drives +  12x Data 200GB Intel 710 SSDs in RAID10 and 384GB RAM

First we set up the first SQL2012 Cluster with those new R720 machines. The new R720s are identical; they became NY-SQL01, NY-SQL02 and OR-SQL01:

• NY-SQL01
  • Primary: Sites DB, Stack Overflow
  • Backups: Sites DB & StackOverflow Full and Transaction Logs -> NY
• NY-SQL02
  • Replica: Sites DB, Stack Overflow, Chat DBs
  • Backups: Chat DBs Full -> NY
• OR-SQL01
  • Primary: Chat DBs
  • Replica: Sites DB, Stack Overflow
  • Backups: Sites SB, StackOverflow & Chat DBs Full -> OR

For this we have 2 new availability groups, StackOverflow_AG and Chat_AG. The primary server for StackOverflow_AG is NY-SQL01 replicated to a secondary in the same data center (NY-SQL02) and across the country to Oregon (OR-SQL01).  The Chat_AG is only on 2 servers: the OR-SQL01 primary (chat is hosted in Oregon) and the replica NY-SQL02. The reason chat is only on 2 servers is because SQL2012 availability groups do not have the ability to distinguish between sites and replicate that way…so it would send the same transaction stream across the country twice to replicate to the NY servers, rather than echoing the transactions through one to the other…this is an unnecessary use of bandwidth we feel.

The StackOverflow and Sites DB portion of the first cluster was completed on the 2012-08-11 maintenance window; chat will be completed on 2012-08-18 (part of chat has moved, we want to give it a week to observe any problems).  Now what happens in the following week? We need to shuffle some hardware around.

With the StackOverflow DB moved off of the NY-DB03 and NY-DB04 pair, they’re ready to be re-tasked.  Currently these servers are identical Dell R710s with 288GB of RAM, 2x OS Drives in a RAID 1 and 6x 200GB Intel 710 SSDs in RAID10.  These boxes get re-tasked to be NY-SQL03 and NY-SQL04. Joining them in this second SQl2012 cluster is OR-SQL02, that new Dell R710 above.  Here’s a breakdown:

• NY-SQL03
  • Primary: All Stack Exchange 2.0 Sites other than SO
  • Backups: SE 2.0s Full + [diff of trans] -> NY
• NY-SQL04
  • Primary: SE 2.0 & SO Dev DBs
  • Replica: All Stack Exchange 2.0 Sites other than SO
• OR-SQL02
  • Primary: Chat Dev DBs
  • Replica: All Stack Exchange 2.0 Sites other than SO
  • SE 2.0s Full + [diff of trans] -> OR

Now we’ve freed up the NY-DB01 and NY-DB02 boxes, they’ll also be nuked, get some new drives and be re-tasked for some other purposes (for example, one goes to Oregon to be the HAProxy traffic log out there).

For the miscellaneous bits, OR-DB01 will be freed up after the move to OR-SQL01 and OR-SQL02 of the chat DBs.   We’ll then take OR-DB01 and install 2012 re-tasking it to host the data.stackexchange.com databases.  It has double the memory of the current server and should provide a nice boost to performance there.

Why? What does all this moving get us?  Well it turns out SQL 2012 Always on Availability Groups give us quite a bit.  Here are the big ones for our architecture:

• Near real-time replicas of every production database, ready to go
• No more copying backups across to the offsite datacenter for redundancy
• We can read from the replicas, eliminating the need for an entire server and allows us to spread the read load out (e.g. API can point at a replica)
• A backup DR location is now doable

First, we can have very near real-time hot spares for all production databases (previously, we had up to 8 hours data loss between differentials).  Second, we don’t need to do these wasteful copies of databases across the country purely for backup purposes…we have a nearly-in-sync replica across the country we can do speedy local backups from.  That’s a huge cross-country VPN bandwidth savings as an added bonus.  Third, we can spread the read load out across multiple servers (and we can add another 2 more to either of these availability groups if needed).  Performance-wise, we don’t even have a need for any read load spreading, but it’s very nice to have as an option. Now for the last one: a DR site.

PEAK Internet in Oregon is where Stack Overflow began on a single server, and we’ve been very happy with the service provided ever since.  Chat’s been all alone out there for over a year now, it deserves some company.  In another blog post coming up I’ll detail how we’re setting up as a read-only disaster recovery location out there, as well as our intention to actively use that while we move datacenters in New York.

P.S. Make sure to RAID your PCIe SSD drives, we’ll put up a post with that story a bit later…

Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.

OODA in Practice

As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.

Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.

Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.

The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.

This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.

Common Problems

This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.

When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.

Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.

If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.

OODA Done Right

Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.

With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.

Why it Matters

If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.

I have to apologize to the serverfault community for a few things:

First, we’ve been really, really busy around the offices of Stack Exchange, and we’ve just not had a good amount of time to write blog posts.  Luckily, now that we’ve added a new Systems Administrator, Mr. Bart Silverstrim (http://serverfault.com/users/13647/bart-silverstrim) we might have some time for posts more often.

Secondly, we’d like to sincerely apologize for taking Bart from the ServerFault community.  I’m sure with all the coffee he’ll be fetching, there will be little time for him to increase his already sizable 25k reputation (and give me time to catch up with him.)

All joking aside, we are very pleased that Bart is joining our team; he’s not only a very smart admin (which his serverfault profile will prove), he’s also a great guy to be around.

So, WELCOME SERVERFAULT VALUED ASSOCIATE #0000004!  May you not crash the blogs with large images.

Bart, modeling his Server Fault tee shirt

In order to make upgrades to our network infrastructure, we will be doing maintenance that will involve intermittent downtime tomorrow. The maintenance will start at 2PM UTC on April 14th, 2012 and will continue for several hours.

During this window we will be:

• Replacing our routers and firewalls
• Making some L3 Network design changes

The long term design change will involve changing our WAN and load balancing infrastruture from Active/Passive failover to Active/Active. When this is completed in the following weeks there will be a detailed post explaining our new design!

I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

• Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
• Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
• Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
• Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

Like any good family, when we saw that our older sibling had something we wanted what did we do? Well, we whined and complained until we got it!

Stack Overflow recently finished their 2011 survey, and as soon as the results where announced, Server Fault being the proper younger sibling that it is wanted one too.

So I call all of you Server Fault users to come and take our survey, you wanted it and now you’ve got it! (oh and you really should get your sysadmin friends to take it too – the more data the better!)

Just like the Stack Overflow survey we will be releasing the data right when the survey ends!