Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.

  • Doug Luxem

    The example you are using is exactly how I would describe the difference between incident management an problem management in ITIL.  In the first, you are simply resolving the problem to get a service back online, but in the second, you are really doing a root cause analysis to understand the problem and probably develop a long-term fix or change to the service.  Its good to understand the difference, and why or when you would perform one or the other.

    Part of bringing professionalism to system administration is having admins that understand and work under a common service management framework (like ITIL or MOF). Otherwise we are just wasting our time reinventing the wheel (or worse, inventing square wheels.)

  • Anonymous

    And when you’re walking through the datacenter, you can always push the red button.  That solves every problem immediately, but you’re only allowed to use the special powers of the red button once per job.

    • Justin Dearing

      If the red button releases the halon, then that’s a once in a lifetime trick.

  • Matt Simmons

    “Press the green button twice” is an excellent example of Cargo Cult System Administration – relevant:

  • card data recovery

    Well, very nice that green always serves for saving purpose, green light, green button as well as green environment, so, serve green and live green.

  • Pingback: On Indian (Residential Internet) Service Providers « nareshovの日記