About a  year ago, I wrote a post about the basics of getting a job in Information Technology.  Unfortunately, that post (linked here) did not cover much relevant material when it comes to putting your resume in and getting an interview.  This blog post has been in the making for a long time, however the major impetus came when a serverfault regular tossed me a message over the weekend asking for suggestions on how to answer “The Salary Question.” I address this below, but first I will cover what I think is the single most important thing to remember in IT when it comes to getting hired.

“It’s not what you know, it’s who you know!”

I have heard many people bemoan the fact that it is near-impossible to get an interview for tech jobs in this market.  Their stories are generally the same, listed below:

• See there is a position at Awesome Company, LLC.
• See that the resume is asking for every technology buzzword imaginable
• Asks for 10 years experience with Windows 8
• Decide to send in your resume for it, even though you only know 3 or 4 of the 15 buzzwords they’ve specified.
• Become depressed when you don’t get a callback.

This is not an unusual turn of events.  The reason behind this oftentimes is that HR is putting together a job description and they’re (optionally!) asking the team that’s hiring for “some technologies they should look for.”  Oftentimes that hiring team says “Well, if they know ABC, XYZ, TLA, ETLA, or Java, it’d be a plus.”   This turns into “applicant must have X years experience in ABC, XYZ, TLA, ETLA and Java.”  HR just screwed not only your chances of getting hired, but the hiring team’s chance at getting a potentially good candidate.

The thing that is tough for most people to understand is that all of the above is what I like to call an “HR Trap.”  It’s there to trap your resume and never get it to the hiring team for consideration.  So, as problem solving IT thinkers, we need to think around this problem.

Do you have any friends that work at Awesome Company, LLC?  Are they friends with the IT guys, per chance?  Ask your friends if they’d be willing to hand your resume to the team that’s doing the hiring along with a recommendation.  (NOTE: Don’t sound needy!  It’s a good way to sour friendships.)  Some people advocate saying “Hi James, I noticed your company is hiring for a position that I’d be great at, and I’d really love to get an interview.  Would you put my resume on Tim The IT Director’s desk for me and possibly put in a good word or two on my behalf?”  This can really work.

Network, Network, Network!

There’s another thing we can do to increase our chances of getting hired in the IT community: go meet people!  This is called “Networking” to the sales world and is critical in getting their job done.  I do know that a lot of us consider ourselves antisocial, but realistically, if all of us “antisocial” types get together, we generally find things to talk about.  Don’t be afraid to approach people, please!  This process only works when you are the one taking initiative.  Here are some options for networking:

• Conferences (LISA, LOPSA-East (nee PICC), Cascadia, Velocity, many others) are great places to network with other people and rub shoulders with the giants of our industry.  For instance, Vint Cerf will be delivering the keynote at LISA this year.  If you ever wanted to meet him, if you were at LISA you could just go up and say hi.
• Local user groups (Like LOPSA presentations, Linux User Groups, Windows User Groups, Tech Meetups) help a great deal as well, and for most of us without a big pocketbook, they’re much cheaper to attend than a conference.  You still get an opportunity to meet a gaggle of new people, and that’s helpful when you’re looking for a job.  I know that LOPSA especially, at least the chapter that I belong to, asks if anyone has job openings they’d like to talk about.  If someone’s asking for something you can do, go talk to them!  You’ve just instantly gotten ahead of the game by meeting the possible stakeholder(s) before HR has had the chance to accidentally delete your e-mailed resume.

Show the world how awesome you are.

An oft overlooked part of the hiring process is the inevitable google search that your potential employer is going to do.  What kind of stuff are they going to find?  Are your Facebook photos from your buddy’s beer pong tournament open to the public?  This is what could be called a “negative public artifact.”  Some tech companies would find it funny, but stuffier business types would consider that an immediate turn off.  On the other hand, if you have a blog full of insightful comments on the industry and diary entries of your (mis)adventures in technology, you show that you’re not simply a piece of paper with a lot of certifications, but someone who actually produces good things.  This is where being a member on a Stack Exchange site is very helpful.  If you mention you’re a serverfault user on your resume and provide the link to your profile, your potential employer has one place to look for all of your questions and answers to judge how well you know your stuff.

Fun With Resumes

I know a lot of us are holding onto the same resume format we had when we started, and are just adding fields as necessary; it’s not always a smart idea.  The job market and what people are looking for on resumes has changed since some of us started looking for jobs.  The emphasis is shifting from “show your experience” to “show how you’re awesome;” if you want your resume to get a second glance, it has to clearly and concisely illustrate why you’re a slam dunk.

1972-1977  Sanford and Son, Salvage

• Drove a salvage truck
• Excellent problem solver

If your resume has sections like the above, you are boring the heck out of the person reading your resume.  Sure, it shows what you literally did in that job, but it doesn’t convey what you’re proud of.  Did you make the salvage equivilent of the Kessel run in less than 12 parsecs?  How is the potential employer going to know about this if you didn’t say so?  Lets try this again:

Sanford and Son, Salvage (1972-1977)

• Frequently noted as having the best one-liners in the company
• Kept a 1952 Ford Pickup in running order much longer than previously considered humanly possible
• Voted the fastest salvager on the east side after completing the Kessel Run in under 12 parsecs! (City record!)

This is not unusual advice.  I’m sure many of us learned it in school when they talked about writing resumes.  At the very least, we’ve spiced up an otherwise dull list of abilities/accomplishments.  Why stop there, we can still do better:

Sanford and Son, Salvage (1972-1977)

Accomplishments

• Ran the Los Angeles Kessel Run in under 12 blocks, beating the standing record of 16 blocks. (1972)

Accolades

• In Trash We Trust Award – 1973
• Salvage Monthly’s “Top of the Heap” – 1975

Skills Utilized

• Advanced engine mechanics and collision repair
• Navigation of South-Central Los Angeles by automobile and commercial trucks
• Lifting heavy loads

If we look at the above, two things are important to note.  First, it has easy to read sections.  Second, it has superfluous space taken up by sections.  If you were to put the whole body of your experience down in this format, you’d have a 4-5 page resume.  Some places might like this, but the common knowledge is 2-3 pages, max.  Without some massaging, your long-winded resume will be too troublesome to read.  Why?  Oftentimes we’re handed a stack of resumes and after a while, it all gets so repetitive that you confuse which applicant had which experience.  The key to resolving this is massaging your work experience to be relevant to the job that you’re applying for.

Tailor that resume, chief

I know that it’s hard work to tailor resumes for every position, but at the very least you should have a few tailored resumes in the areas that you are most proficient.  For example, I have three separate resumes that I give out if I’m applying for a developer, IT, or management position.  I’ve done all three jobs, so I have three tailored sets of job experience that subtly (or perhaps not-so-subtly) emphasize those roles in the full body of my experience.  Sometimes I leave a job out that isn’t relevant to the position, but I’ve found some interviewers are taken aback when you mention you left out some job experience.  Why would  you care if I slung lattes at a coffee bar while in high school, I ask?

Cover letters are mandatory

You might have heard people say that cover letters are dying out in this age of digital resumes.  They’re lying so they can take your jobs.  Every single resume you sent out needs a cover letter, and it needs to be customized to the company you want to work for.  I know that this is a tall order for some, but if you’re a prudent applicant you’ve already studied the company you’re applying for and know what they do or provide.  Why not explain to the person who’s going to hire you why they should waste their day interviewing you?  The cover letter is your “First Impression” on paper.  It helps to show how eloquently you convey yourself on paper, and it’s  pretty important in tehnology jobs because a lot of our jobs involve some level of technical writing.  Ignore this advice at your peril.

Please provide your starting salary…

Yeah, if you want to get priced out of the market.  The answer to that question should always be “we can negotiate that after the interview when you’re sure I’ll be a great fit!”  If they press, try to dodge as much as possible.  The salary question allows them to automatically throw out resumes.  If you come in low, they think you’re not in touch with the job requirements or the job market.  If you come in high, they’ll think you’re either presumptuous or literally are too experienced and therefore too expensive to hire.

Your resume will give them an accurate idea of how much they should be paying you just by the virtue of how much experience you’re showing.  If you’d like more details on this, check out this year’s Robert Half Salary Guide.  This gives an accurate starting point to figure out how much money you should be asking for or how much they should be paying you.  However, don’t tell them this number (or your imaginary What-Am-I-Worth-A-Year number) until after they’ve decided they want to hire you.

Closing thoughts

I don’t want to end this on a somber note, but please keep in mind that if you’re trying to get interviews at companies where you have no inroads or networking contacts, please be ready to get a callback from 1 out of every 20 or more resumes you send.  It is a cut-throat job market and there are hundreds if not thousands of people applying for the same positions you are, but they’re better at bullshitting on resumes than you are.  It’s tough to get a job without having friends in the industry already.  Don’t get discouraged!  Do what I advise above, especially on the socializing front, and you’ll be much better off.

Appendix 1: What a well-formed Systems Administrator job reference might look like

I was asked to provide an actual SysAdmin-focused job description to help some people along.  Below is a completely fictitious entry that I’ve generated off the top of my head.

Senior Systems Administrator, ImaginedCompany, LLC (2008-Present)

Relevant Technologies

• Advanced Active Directory provisioning and maintenance for a 100,000 user company
• Deployment automation via Windows Deployment Services in Windows and KickStart in Linux
• vSphere setup, troubleshooting and administration with multiple Datacenters and Hosts
• Deep Telephony – PRI, TDM, FXO/FXS setup and maintenance
• Configuration Automation via Puppet and Systems Center
• Revision control with Mercurial, Git and Subversion

Accomplishments

• Independently developed the Windows Deployment Services installation for ImaginedCompany, decreasing provisioning time from hours to minutes.
• Converted legacy servers from bare-metal hardware to vmware, saving the company thousands of dollars a year in electricity
• Quickly reimplemented a PBX solution when the legacy telephone system hardware failed, enabling the company to continue to make money until a permanent solution was in place.
• Led-by-example with new project to use source and configuration management by way of Mercurial and Puppet/Systems Center

Accolades

• 2008 – LOPSA Most Active Mentor
• 2009 – Golden Baseball Bat recipient for best client-facing customer service.
• 2011 – Employee of the Year

It is pretty hard to get a bunch of system administrators together for any period of time and not have a conversation about command line for managing things. In my experience, command line always wins the debate for any medium to large installation. Microsoft caught on to this and has introduced powershell. From my own experience and people I have talked to, powershell adoption has been slow. Like powershell or dislike it, it is an interesting take on command line with its object oriented versus text-based approach. It is also quite powerful, so why the slow adoption?

In my mind it isn’t a problem with powershell itself, but rather a result of human nature, culture, and the Windows ecosystem. The problem lies in workflow, and I think the best way to illustrate this is to contrast Windows versus Linux administration. When managing Linux systems, scripting something flows naturally from the experience of trying things out in the first place. For example, lets say someone with some Linux experience (but not an expert) is building a piece of software and installing it on Linux for the very first time. The steps usually involve something like:

1. Run the traditional, configure, make, make install from the command line. See what libraries or binaries are missing that are required to build, from the errors. Install them (often via yum or apt), keep doing this until it works.
2. Possibly add some users to the system, adjust permissions etc with commands like useradd, chown, chmod.
3. Edit some text files
4. Set it to start at boot with commands like chkconfig or update-rc.d

As you do this more and more, you learn the patterns of what fails, and get the urge to automate it. Although things like puppet are the end goal these days, the next normal step is to write shell scripts to do this. This is the lynchpin of the whole experience, a shell script is just a little flow control, error handling, and a few variables on top of what you were already doing. With the Linux experience, scripting flows naturally as a next step in your experience. It is a gradual learning experience, and your previous experience of doing things manually taught you much of what you need to know to script it — what goes wrong, what to check, how to do everything you need to do. Even the first time tutorials on the web are almost always command line. From the second you start learning how to administrate a Linux box, you are learning how to script.

Unless you are extremely disciplined or have been taught Windows administration recently by a “benevolent” dictator, your first steps in Windows administration are via the GUI and Wizards. Don’t get me wrong, for many things this gets the job done faster and with far more ease. The generally accepted problem with the GUI though is that it doesn’t scale. When you want to scale in management you often turn to scripts (GPOs are very useful, but you are generally out of luck if what you want isn’t predefined.) With Windows, scripting these tasks is generally a complete departure from your previous experience. Scripting doesn’t flow naturally from previous experience, and scripting something is scary because of this. This means the windows ecosystem, due to human nature, is far less likely to produce administrators with scripting experience. Therefore, powershell adoption is logically slower. This also means administrators are less likely to be testing powershell and pressuring each other and Microsoft to improve it.

The cure to this workflow to me seems to be Windows Core since changing human nature isn’t easy. With Windows core, even though you can use the Management Console, you really have to start doing many things via the command line and powershell. With use of these tools, just as with the shell in Linux, scripting will flow naturally. The Catch-22 however though is that since adoption is slow, everything you need might not be available via command line options. At my company we haven’t used Windows core, but I’m hoping with Server 2012 it is time for Core, does anyone with more experience know if it is time for this shift?

Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.

OODA in Practice

As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.

Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.

Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.

The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.

This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.

Common Problems

This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.

When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.

Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.

If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.

OODA Done Right

Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.

With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.

Why it Matters

If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.

I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

• Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
• Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
• Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
• Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

• Placing a value on performance: “Performance is a feature”
• Well integrated development and operations
• A sense of craftsmanship when it comes to performance

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

• More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
• Licensing Costs
• Owned vs Rented Assets
• How many cloud “servers” or instances you would need vs real hardware
• Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.

One website defines production control as the following:

Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.

That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.

Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.

Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.

As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.

An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.

For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.

Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”

The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.

Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?

I have always had the notion that some companies “Get It” when marketing to programmers, sysadmins, and hackers. A couple weeks ago I was in Silicon Valley at Tech Field Day listening to a lot of presentations with some fellow geeks. In some of these presentations they “got it”, in others they didn’t.

The fundamental difference is that the ones that get it tell me about their technology, and the ones that don’t tell me about their products. True geeks have one common trait, and that is that they like to learn how things work. We get excited about learning and news ideas. Exciting technologies solve problems in non-obvious ways. Since these are not obvious, we want to learn about them, and in as much detail as possible.

Products, on the other hand, are how these ideas are packaged. For the most part, we don’t care. If we do care, that comes later, only if it will solve a problem that we have will we start to care about implementation and packaging. The advantage of telling us about your technology is that we do care, even if we don’t have a use for it at the moment, because we are geeks – it is our nature. When done right, we will then associate your product with the technology and that will be enough.

This has some consequences that marketing should be aware of when targeting tech people. First, we generally don’t want to talk to you directly, at least, not for very long. This is not because you are not important or interesting, it is just that you probably can’t get us excited about your technology like your engineers can. You can enable your engineers to present your technology well, and that is what people who are good at geek marketing do. If you can’t get your actual engineers to present no mater how hard you try, your screwed. It’s not your fault, your company just sucks and you probably just need to move on. Lastly, everything targeted to tech people should be aimed at getting us excited about your technology, not your product.

With my recent experiences at tech field day there were some good examples of this done right. Pure Storage taught me about why I see SSDs fail and a new type of RAID they invented suited for SSDs. Arkeia educated me about various implementations of deduplication in backups. Data Direct Networks introduced me to the concept of object store filesystems. However, in that case I wanted to learn even more given that amount of time. How successful they were came down to how much time they all devoted to fulfilling my desire to learn more about technology vs. telling me about their products.

Our CEO Joel Spolsky wrote a book about hiring programmers that is titled Smart & Gets Things Done. The idea behind the title is that when it comes down to it, you really just want to hire someone who is both smart and will get things done:

“People who are smart, but don’t get things done … would rather mull over something academic about a problem than ship on time”.

This applies equally in the area of system administration. Our CTO Jeff Atwood recently tweeted a reference to this:

When I read that I felt like he was talking to the sysadmin team (I hope he wasn’t — but the fact that I felt like he might have been is enough). The reason for this isn’t that we are not getting things done; I think we work hard and get a lot of things done. Rather, I think we have tendency to pick the interesting tasks over the uninteresting or tedious tasks.

Assuming as a system administrator you actually like what you do, I think tasks end up looking like the following:

So there are some interesting tasks that would be good to do, but may not need to be done. Then you have most tasks, which if you like your job, both need to be done and are interesting. Lastly, you have the tasks that are not interesting but do need to be done.

With a tendency to give too much weight to the interesting tasks or pieces of a project, that category on the right will grow over time. I like to think of keeping that category under control as eating my vegetables. Ideally, one maintains a good balanced diet all the time. But in reality there are times when you need to get on the scale and come to grips with the fact that you might need to go on a healthy diet for a while.

Lastly, don’t worry, the irony of taking some time to write this post is not lost on me.