I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

  • Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
  • Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
  • Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
  • Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

  • I don’t know if it’s necessarily a sysadmin’s fault in many cases; I know there are plenty of times where I feel pressured into taking shortcuts because certain users require it be done nownownow rather than a desire to do something in a half-arsed manner.

    • Anonymous

       I think there can be a distinction between scope and then being thorough within that scope. There are frequently constraints in terms of money or time (or whatever else) in all positions that can limit the scope. But if someone is thorough in their work, they make the most of what can be done within those constraints.

      •  This is true; there are situations where you’re handed a crap situation and you just do the best you can within that situation. My case in point is working in a public school system where we have to come up with “Creative Solutions” (i.e., as free/inexpensive as possible) and started virtualizing hardware. Corporate IT people cringe to hear we were running one ESXi host knowing that a non-RAID type failure would knock out multiple servers, but we did it; I was being diligent in making sure we had the VM’s backed up through our normal Windows backup routine (for Windows servers) and by shutting them down periodically to get compressed images of the machines saved to another location as archives, and if need be I had a plan in place for firing up the VM’s on individual workstations if the host server died. It’ was a horrible hack solution, I knew it, but it’s the best we could get with budgeting constraints.

        We finally got a second server (after much hounding) so we could have some images running on one and some on the other in case a host died. Unfortunately they’re not feature-parity with each other because of how long it took to get the other host, but again…doing what we can.

        My primary issue is when it becomes pervasive to just get a solution thrown in place and left alone. There are situations where departments play SURPRISE WE ORDERED THIS AND NEED IT INSTALLED ASAP. I think there’s a point where hacking a solution together switches from a necessary balance of constraints in a situation to a cultural value in an organization. The former is frustrating. The latter is demoralizing for people taking pride in their work, or damaging if you come into the organization new and think that’s how it’s supposed to be done.

    • Michael Graziano

      Sometimes a half-assed solution solving a critical problem is better than waiting until a properly architected solution can be built, provided the half-assery is thoroughly implemented.

      Case in point, 3 years ago I hacked together some perl one-liners to provide our billing team with customer usage figures so they could get invoices out more quickly. Prior to this they were opening up our software and manually counting activity for each client.  The resulting system is a wonderful example of tempermanence in action — The solution was temporary (I expected it to last a year at most, and then we would do a proper integrated billing system), but to date business constraints have kept us from coming back to that project, so the temporary solution lives on.

      Should we have stopped everything and written that integration layer? Correctness says yes, business demands say “We can’t spend 3 months on this!” – Business demands win, and the half-assed improvement makes life a little less painful for a department that already has too much work.

      The other side of the coin is that I took my time writing and debugging the code, fully knowing that “temporary” often becomes tempermanent, and that the business would be relying on this code heavily once it was rolled out. It’s a half-assed solution, but a “whole-assed” implementation, with protective plastic over all the edge and corner cases.

  • I found the point regarding felling extremely overwhelmed very relevant to my own case. Great article to read.

  • Since IT can be significantly capital intensive, one of the sources of bandaid-itis and slog-contribution can be needed upgrades/improvements put off for better revenue-weather. At my old job, which is now entering Year 4 of an Ongoing Budget Crisis, the problem was getting acute when I left a year ago and is probably agonizing by now. You can’t forklift-upgrade a $120,000 system when there is no money to be found, even if the vendor is EOLing the device.

    The same kind of thing holds true for revenue-driven business.

    “My that’s a big dollar figure, next quarter. Maybe.”

    Even though you’ve spent three months making the case that this is a needed expense for risk-management reasons. 

    “We’ve survived this long without it, we can last another three months.”

    It’s the sense of powerlessness to meaningfully affect your environment that really drags on you after a while. You can polish all the rough edges, post warning signs around the trip hazards and install guard-rails over steep drops where you can get away with it, but that isn’t the same as removing those items from the environment.

    The standby answer to this has always been, “GET A NEW JOB, BOZO!” That’s not possible for everyone so better coping skills.

    I’ve done everything I can with the resources I have available to me to make sure things are running smoothly, predictably, and in a documented fashion. But further progress is blocked on the management layer. $SlogFactor=$SlogFactor+10.

    The only thing for it is to improve revenue somehow (rarely doable for IT, but not impossible), or become better at persuasion.

    • Temporary

       A coping mechanism could be to not give a damn if things go bad bad, to make sure you have properly documented your ignored efforts to avoid the disaster and to be sure to put the blame where it belongs (in the hope they will be let go). I know that sounds rather negative, but what would be a better option if you can’t change jobs and people won’t listen?

      • That was the exact mechanism chosen by one of my coworkers at my old job. Then I left. Then I hear he left four months after I proved it could be done.

        He was bitter and hard to work with towards the end, and I don’t wanna end up that way. But yes, that:

        Screw it, I’ll just save up for one big I Told You So when the ball drops.

        temptation is big. The big hazard with that route is that it’s all too easy to stop caring about being thorough in everything else you do have control over.

        • Temporary

           Yeah I agree. There’s just no ideal way to cope with a situation that’s so messed up. It’s quite a shame so many people are ignorant about the importance of keeping things up to date and all that.

          • If management refuses to be proactive, sometimes you just have to develop your contingincy plans on your own and wait for things to fail. When it becomes an emergency, then you’re likely to get all the budget you ever asked for… maybe more.

  • Andomar

    Always plan ahead? YAGNI.Embrace criticism? Breeds an air of negativity.Emphasize quality? Quality is always a tradeoff.

    Worst serverfault blog post ever.

    • Lucas_kauffman

      You sir are a coward and a ruffian!

    • Reall Now

      You aint’ gunna need PLANS? Are you insane?

      Embracing criticism is not the same as promoting criticism, and proper criticism doesn’t result in negativity. Few things that aren’t critiqued and improved are very good.

      Quality being a tradeoff doesn’t mean you can prioritize quality.

      Worst thought out comment ever.

    •  How can you not plan ahead for potential problems? When something goes wrong do you just run around in a panic? If not, then you must have some form of plan.

      Embracing criticism is probably the best way I’ve found of finding flaws in ideas. Constructive criticism is a key part of growing and learning. Negativity breeds when you shoot down ideas without any explanation why something is flawed and lack a proposed solution or modification to fix the problem.

      And quality is a tradeoff for the given situation. That’s no excuse for not doing the best you can with what you have.

      Judging from just this comment it sounds like your job situation has made you bitter or maybe need a bit of a vacation…

  • When I’m working with someone who is new to the field, one of the first things I instruct them over is how nothing is ever temporary. I know of one multi-mega-million pharmaceutical company who has a large amount of very important financial information that is processed in a frankensteined Excel spreadsheet that a lowly data steward hacked together to automate some mundane tasks.

    Within months it had a few features added in that a manager wanted. Within I believe 18 months it was processing large amounts of finances and there was now a departmental war over who would take the spreadsheet over to fold its capabilities into other systems.

    Nothing is temporary.

    • Temporary

      Actually, nothing is permanent and everything is temporary. And one’s half assed stop gap fix may confirm that by causing the company to become very temporary and go belly over. 🙂

      • Yeah, I saw the paradox in my statement. How about: Everything is temporarily permanent?

        • Michael Graziano

          Design all systems as if they will still be in service in 100 years.

          If you don’t have time for that design, at least ensure that you have accounted for the 5- and 10-year failures, because invariably that 6-month “temporary” bridge will become a multi-year “tempermanent” fix when construction of the permanent replacement is stalled.

          • Temporary

             I agree with the idea. However I think looking beyond 10 years is a bit too much. Unless you’re using mainframes it’s likely the hardware has either broken or has been replaced. With that in mind add to your design goals easiness of migrating to a new system.

          • Yeah. That’s what they said about all those programs written for mini-computers that are still running 40 years later too. I use nothing but cast off old PCs and even my 15yo 400MHz PII Micron can run a website. I do my best to write code that will run forever (yeah, that means using the FULL leap year algorithm… which is only about 10 lines anyhow).

        • Temporary

           Very well 😉

  • Florian Heigl

    This is a brilliant article. There’s far too little literature on how / why good systems administration practices are achieved. Everyone will get the usual warning against doing rm -r (-f on toy OS), but hardly anyone is also taking the time to teach people just how to safely delete. 

    At some point I started writing down small manuals for routine tasks. It was just intended for a friend.

    You can have a look at the articles in the wiki below. and if you wanna pass them on, please do so or like the idea and wanna join, let me know. :> http://confluence.wartungsfenster.de/display/Adminspace/Pre-reboot+checks

    •  To be fair there are good practices then there are good practices for particular organizations and their situations. Maybe it’s better to have best practices for the situation and an ideal practice if you had the budget 🙂

  • Staticsan

    Users “could care less” about downtime if they know it’s coming? I really don’t know what you’re trying to say here. In my experience, users don’t like any downtime, scheduled or not. Arranging for a maintenance window is usually an exercise in horse-trading and you rarely get what you need, let alone what you want. 

    (And I wasn’t sure I really wanted to comment when I saw you were using Disqus…)

    • Anonymous

       Hello!  First off, thanks for the feedback on disqus.  We use it for all of our blogs at Stack Exchange.  I’m not sure what the motivation was for using it, but it’d probably take an act of congress to change it.  Sorry this is a friction item for you.

      Regarding the downtime comments: this position assumes that the downtimes you are choosing are what the industry would consider “standard”, i.e. after-hours or weekend downtimes.  Normal business hours are generally not valid for planned maintenance.  The majority of end users will not be using the systems and because of this, generally don’t care unless they were intending to take work home with them.

      I know this doesn’t apply in 24/7 operations, as my previous employer was a 24/7 shop replete with procedure and rigmarole regarding downtimes.  For a downtime there, you needed to file a change control task two weeks in advance with rationale for the downtime.  In these environments it can be harder to violate the downtime window;  for me, if I realized mid downtime that something we had to do was going to take the window out of compliance, we’d abort and revert, then wait the extra two weeks to get a larger downtime.  Sounds insane, I know, but it’s either do it right or get your butt handed to you later on when the hack-job takes down a client’s business.

  • Travis

    I share your opinions and think you did an outstanding job of expressing them;  however you may want to consider a balancing this with an emphasis on clear objectives, productivity and results. I’ve successfully demonstrated and conveyed this quality/thoroughness point to the extent that my team members often struggle to understand how to take a task to a level they haven’t operated at or how much is going to be enough. This isn’t a terrible position to be in until their productivity drops through the floor while their straining to measure up and their increasingly frustrated while their confidence diminishes. The focus on objectives and results helps guide them while providing an action plan.

    This article could be split into multiple addressing both the importance of mentoring the junior admin and separately addressing the people that should know better. Often, the junior admin problem is not knowing what they don’t know and insufficient experience. They need a good mentor. The poeple who should know better could be admins, programmers, PM’ etc, but if they don’t understand the value of business level work, they need to be find a new profession.

    BTW – Still in Pottstown?  I worked there years ago and spent many years in the surrounding area. 

  • Marc Jellinek

    Time for some truisms:

    If you fail to plan, you plan to fail.

    There’s never enough time or money to do it right.  There’s always plenty of time and money to do it twice.

    I’ve hear the argument that time-to-market beats all, and the next version will be here soon enough.  My reply has always been:  you only have one shot at making a first impression.

    If it’s not important enough to do right, why do it at all?

    Last but not least:  picking the lesser of two evils is still picking evil.