A question that comes up again and again in web development companies is:

“Should the developers have access to the production environment, and if they do, to what extent?”

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers — so please don’t take it this way. First I want to cover a few common arguments of developers that dislike or hate this idea:

“We can’t get stuff done, the system administrators get in the way and take forever.” Well if this is actually the case, then they are right. If there are not enough administrators or the administrators are not good then they can become a bottleneck. However, developer access is not the solution because after this you still have crappy or not enough administrators. Sometimes there are other administrative specific concerns that might make things take longer, more on this later, but it shouldn’t take an unreasonable amount of time.

“We have always had access before.” Startup companies seem to rarely start out with administrators. For some reason system administrators are considered a luxury. Although this process might have worked before, as you grow there is probably more administration. Things get more complicated and this is probably why they went out and hired an administrator. So in this case, “this is what we have always done” isn’t really good enough argument.

“We need access to troubleshoot.” Maybe, maybe not. It is possible the administrators can just give you the information you need. If this particular area becomes a bottleneck, limited access might be in order.

Those are a few possible arguments against restricted access for developers, but lets move on to what I really want to talk about — why it is a good idea.

The Process Restricted Access Creates: If the developers can not access production one big implication is that they can’t install their own code. This means that the administrators must install the code. Two things then need to happen: 1) The developers and system administrators must communicate — with each other! The administrators learn how to install the software which I hope I don’t have to explain is probably a good thing. 2) The developers have to create an install or release process that is easy and effective. This is also a good idea. Being able to rebuild the environment is an essential part of disaster recovery. So what can’t happen with restricted access is that the installation of the code is some complex process that only lives in heads of a few developers. Also, the developers don’t have to spend time deploying and installing code when they could be writing new code. It might take them longer at first, but asymptotically this is will be faster (That is right, I used a fancy developer word).

Administrators will also probably learn a little bit more about what needs to be backed up through this process. If the administrator doesn’t know the application well they just have to trust that what the developer told them to back up is all that really needs to be backed up.

Developer’s Concerns are Often Not System Administrator’s Concerns: In general developers do not focus on security in the same areas that system administrators do. They usually have different areas of expertise when it comes to web site security. Topics such as cross site scripting and SQL injection are likely areas of security where developers have specific expertise and administrators do not. Account privileges, file permissions, web server configuration are often not what developers have experience in or are very interested in. These are all important areas in production environments are meant to the expertise of system administrators. Also while I am on the topic of security the less people with access the better (Principle of Least Privilege). This also helps when the call comes in at 2am because the system administrator doesn’t have to wonder if one of the 15 developers with access were on the system doing … something.

Change Control: I don’t think there is a decent developer out there that isn’t serious about change control. When it comes to their code. However, I haven’t seen to many developers that are serious about logging every single change they make to server as a whole (I have seen some configuration files under revision control however).

If this isn’t done it means that the production environment will not be able to be rebuilt properly. It also means that if there have been changes that might have caused a problem those changes might not be know to the person trying to solve the problem. The parallel to this would be if an administrator just went into the production code and changed some things without telling anyone or checking it in. Ya, the developers would freak.

The Person Who Owns it Should Have Control: One of Joel’s Spolsky’s beliefs when it comes to management is:

“Everybody owns some area. When they own it, they own it. If a manager, or anybody else, wants to provide input into how that area is managed, they have to convince the owner. The owner has final say.”

System administrators are generally considered to own the production environment. The administrators are the ones who keep track of uptime, the ones who get the phone calls at 2am, basically, they are the ones closest to the problem. When developers have direct access to production from what I have seen this control always gets undermined.

The System Administrators Responsibilities: In order for this to work, administrators have duties that must be fulfilled. 1) Invite the developers to request what they need from you and be pleasant about giving it to them. 2) Make sure the developers have a good development environment in which they have free rein. 3) Be reasonable and practical. Every company is different, for some companies maybe developers should just have no access because of the nature of the business (i.e. finance). However, if you are not a financial company, a work flow where developers have unprivileged access is likely the best solution. There might also be some developers that double as system administrators so every company has a different situation.

As I stated in the beginning my belief in the process doesn’t have to do with having great or not-so-great developers — many developers function great as system administrators (For example see this post on sending email without it being tagged as spam). Rather this is about a process that lets both people focus on their expertise as a company grows. Things may move a little bit slower. However, the trade in should be that you get a more reliable and secure production environment.

«
»
  • sana
  • Lawrence

    Brilliant post. I am a security analyst for a 50 person company and wondering how to address this issue. I found this post very interesting. Keep up the good work Man

  • I think the answer to this depends on your answer to a couple other questions:

    1. How big is your company? (Do you have the time & resources to dedicate a QA team & Sysadmin/DevOps to managing production & deployments?)
    2. Is the developer culture centered around quality & stability of production?
    3. Do your developers have the time, expertise, and discipline to not make changes to production which are one-off? (Changes made by Engineering In Place (EIP) / hotfixes should be fed back into the codebase & normal release cycle)