One website defines production control as the following:

Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.

That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.

Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.

Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.

As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.

An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.

For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.

Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”

The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.

Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?

«
»
  • Thank you for a very informed blog, this information is great I will definitely be adding you to my favorites list and review these often.  

    • Peter Grace

      Thank you for taking the time to read it!  I’ve got over a decade of experience in IT, some of it in the trenches and some of it in management roles, so I hope to cover a diverse set of topics that I hope will be interesting and informative.

  • It is amazing to see this post these days where process frameworks that are implementing what you call “Production Control” exist since several decades and way through their first reviews.I am speaking of ITIL (IT Infrastructure Library) and associated ITSM (IT Service Management). Also, OSGI is around with a slightly different approach. I do appreciate your thoughts (and can only subscribe and support them), it is yet stunning to see how ignorant the dev world has been and obviously still is about what other IT engineers are practicing since years. Your before-last paragraph actually paraphrases the contribution that development should make: Controlled processes to manage change efficiently: Release Cycles with limited scope (call it a RUP Iteration, an Agile Sprint, …), QA function, Releasing Function. DEV anarchy does not help service stability as you rightly line out. And I thnk that is what most DEVOPS followers these days begin to understand: Managing a service that is dependant on a group of developers (if not a thousand) is hard work and some few even recognize that there’s a herd of engineers already in the field that have learned and documented their experience in such process frameworks. I think it is time to tear down the fence and give you DEV guys a tour around. Just because EC2 is at your fingertips, doesn’t mean you need to invent everything from scratch.

    • Peter Grace

      It always surprises me when I come to a new shop and find they don’t have a procedure in place to handle this.  It seems like a daunting task if you’re not used to doing it, but I’ve yet to hear of anyone actually having a process such as the one outlined above causing an adverse effect on their uptime/revenue.

  • Starcom2002

    kompresor

  • Starcom2002

    kompresor ile ilgili sorunlar kompressor http://www.elkomkompresor.com

  • It’s very successfull article. I’m working as IT manager at a company. I will follow your articles.