One website defines production control as the following:

Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.

That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.

Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.

Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.

As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.

An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.

For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.

Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”

The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.

Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?

I have always had the notion that some companies “Get It” when marketing to programmers, sysadmins, and hackers. A couple weeks ago I was in Silicon Valley at Tech Field Day listening to a lot of presentations with some fellow geeks. In some of these presentations they “got it”, in others they didn’t.

The fundamental difference is that the ones that get it tell me about their technology, and the ones that don’t tell me about their products. True geeks have one common trait, and that is that they like to learn how things work. We get excited about learning and news ideas. Exciting technologies solve problems in non-obvious ways. Since these are not obvious, we want to learn about them, and in as much detail as possible.

Products, on the other hand, are how these ideas are packaged. For the most part, we don’t care. If we do care, that comes later, only if it will solve a problem that we have will we start to care about implementation and packaging. The advantage of telling us about your technology is that we do care, even if we don’t have a use for it at the moment, because we are geeks – it is our nature. When done right, we will then associate your product with the technology and that will be enough.

This has some consequences that marketing should be aware of when targeting tech people. First, we generally don’t want to talk to you directly, at least, not for very long. This is not because you are not important or interesting, it is just that you probably can’t get us excited about your technology like your engineers can. You can enable your engineers to present your technology well, and that is what people who are good at geek marketing do. If you can’t get your actual engineers to present no mater how hard you try, your screwed. It’s not your fault, your company just sucks and you probably just need to move on. Lastly, everything targeted to tech people should be aimed at getting us excited about your technology, not your product.

With my recent experiences at tech field day there were some good examples of this done right. Pure Storage taught me about why I see SSDs fail and a new type of RAID they invented suited for SSDs. Arkeia educated me about various implementations of deduplication in backups. Data Direct Networks introduced me to the concept of object store filesystems. However, in that case I wanted to learn even more given that amount of time. How successful they were came down to how much time they all devoted to fulfilling my desire to learn more about technology vs. telling me about their products.

Our CEO Joel Spolsky wrote a book about hiring programmers that is titled Smart & Gets Things Done. The idea behind the title is that when it comes down to it, you really just want to hire someone who is both smart and will get things done:

“People who are smart, but don’t get things done … would rather mull over something academic about a problem than ship on time”.

This applies equally in the area of system administration. Our CTO Jeff Atwood recently tweeted a reference to this:

When I read that I felt like he was talking to the sysadmin team (I hope he wasn’t — but the fact that I felt like he might have been is enough). The reason for this isn’t that we are not getting things done; I think we work hard and get a lot of things done. Rather, I think we have tendency to pick the interesting tasks over the uninteresting or tedious tasks.

Assuming as a system administrator you actually like what you do, I think tasks end up looking like the following:

So there are some interesting tasks that would be good to do, but may not need to be done. Then you have most tasks, which if you like your job, both need to be done and are interesting. Lastly, you have the tasks that are not interesting but do need to be done.

With a tendency to give too much weight to the interesting tasks or pieces of a project, that category on the right will grow over time. I like to think of keeping that category under control as eating my vegetables. Ideally, one maintains a good balanced diet all the time. But in reality there are times when you need to get on the scale and come to grips with the fact that you might need to go on a healthy diet for a while.

Lastly, don’t worry, the irony of taking some time to write this post is not lost on me.

Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.