At a recent talk at PICC, part of my introduction was:

“Hello and thank you for coming to our talk. This is George Beech and I am Kyle Brandt, and we are the two sysadmins at Stack Exchange…”

While “two” is the technically correct answer going by our self-anointed titles, it isn’t really the whole picture.

The Sysadmin Continuum

In reality here at Stack Exchange we all wear a lot of hats. While George and I may own the system administration tasks, we work together with our developers who are pretty accomplished sysadmins in their own right. If it were just George and I soely doing the work that could be considered system administrator work, we would probably have a pretty tough time of it at only two people.

In reality we are continuum. System administration is just a spot on the continuum where George and I spend most of our time. However, a lot of our developers spend a lot of time there and travel the same road.

Some recent examples:

  • Jarrod Dixon worked with me on developing a log daemon for inserting HAProxy syslog data into SQL Server
  • Nick Craver frequently hops on Dell and makes hardware recommendations
  • Jeff Atwood helps us with SSD research
  • Sam Saffron works on SQL restore scripts
  • Geoff Dalgas racks servers out in OR and troubleshoots our NAS
  • Ben Dumke and Jeff Atwood do some CDN research
  • Jeff Szczepanski, our Vice President of Products, analyzes packet dumps

Because of the spirit of shared responsibility, we get to take advantage of everyone’s knowledge and experience and we are all better for it.

So in the end, “How Many System Administrators does Stack Exchange Have?” is a question I can’t really answer.

«
»
  1. Andy says:

    Does that mean the whole team have full acess to all infrastructure and servers, or are load balancers etc “sysadmin only”?

  2. SysAdmin1138 says:

    I know how this works… before I got to the present gig the SA role was held variously by a lot of people. Right now I’m primary, but it also means that the ‘operational point of view’ is held by several of our developer-oriented people and that’s not a bad thing. DevOps indeed.

  3. Matt says:

    SSD research? SSD drives are so much faster than traditional hard drives that’d probably doesn’t make sense to do too much research. you will see a big benefit regardless. I know jeff is an extreme hardware enthusiast but I gotta think time is better spent. programming.

  4. Kyle Brandt says:

    Hi Andy,

    The load balancers are accessible by the sysadmins and a couple of the devs — but not all the devs.  

  5. Kyle Brandt says:

    It is more about the failure rates of the different MLC vs SLC technologies and brands. Which also vary depending on the generation, before the SLCs had better lifetimes. Now the latest MLCs for Intel are what we believe is the best bet.

    http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid-state-drive-scale.html

  6. Sysadmin Sunday #31 « Boxed Ice Blog says:

    [...] How Many System Administrators does Stack Exchange Have? [...]

  7. Small Business IT Support says:

    Yes you are right…Very nice communication between all.

Leave a Reply