Keeping Your Cool

Steve Murawski

I’m not talking about data center cooling here…

I was recently listening to The Ship Show podcast titled “Keep calm and PROD on”.  In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.

The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents.  The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.

The Scenarios

The hosts outlined a couple of scenarios for discussion.  They didn’t use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..).  In no particular order (going from memory and making up the classifications here), we have the “cowboy” response, the “neophyte” response, and the “deer-in-the-headlights” response.  I’ll provide a quick synopsis of these responses.

The “Cowboy” Response

This incident response archetype is very identifiable in many environments.  Picture it, the year is 2010, it’s 2 AM and your cell phone is blowing up with monitoring alerts.  The website is down and everyone is waking up.  You log in to your VPN and start gathering information.  Everything goes dark, and then everything comes back up, seemingly magically fixed.  It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.

This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I’ve observed over my years in IT.

The “Neophyte” Response

This incident response archetype is your typical “newbie”.  The neophyte does not have to be new to the field, just new to high pressure outage scenarios.  Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up.  The neophyte might not  be comfortable with getting a more senior engineer out of bed, or getting some developers involved.  Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.

The “Deer-In-The-Headlights” Response

The last one I remember the podcast covering was the “deer-in-the-headlights” response, where regardless of a person’s experience level, they just don’t respond well in a crisis.  Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders.  This leads to a fractured response, where people may be working at cross purposes.

What Should We Do?

What I’ve Learned In Previous Careers

Training To Be A Cop

You may not know this about me, but before I got into IT, I was training to be a cop.  I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns.  A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations.  (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)

In that training, we drilled how to respond to emergencies as individuals and as part of a team.  Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes.  Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions.  Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.

Part of the training regimen includes working past the failures and mistakes.  In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we’ve drilled in the classroom and on the range.  We are coached to get to cover, return fire, and call for backup.  We can’t dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.

Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits.  In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for.  We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera.  This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills.  Training helps minimize the “deer-in-the-headlights” and the “neophyte” responses and allows people who would experience those responses to fall back on training.

Working In A Public Safety Agency

I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department.  Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations.  I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.

Under the NIMS model, the first responding public safety officer is the incident commander.  As the situation develops, the incident commander role can change, based on who’s best suited to deal with the incident.  For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command.  (In my experience, this is after police officers have saved all the people in the building. ;) ).  Let’s look at another scenario: officers respond to a report of a burglary in progress.  The first officer responding is the incident commander.   As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation.  Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.

NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer.  The PIO’s responsibility is to keep the stakeholders and public in general informed as to what is going on.  Part of the role is determining what information is helpful to share and what should not be disclosed.  The Safety Officer monitors conditions and ensures the safety of all incident personnel.  The Liaison Officer is responsible for dealing with all the coordinating agencies.  Defining these roles can help deal with the “cowboy” response.  If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus “just get x involved, he fixed it last time.”

How We Should Respond

These two experiences provide some basic thoughts into how we can approach incident response.

Drilling

Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn’t immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.

For operations personnel (developers, dba’s – yes them too – and sysadmins), this is critical.

For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel.  Guess what?  The knowledgeable guy isn’t always around when things go ill.

Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital.  Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally.  Quick – what do you do when your CDN stops serving content?  Have you drilled that scenario?  If you haven’t where do you even begin?

This is something we are going to be focusing more on here at Stack Exchange and I’m super excited about that.  We’ve decided that this is a priority for our organization and we’ll be dedicating time to this.

Roles

In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations.  The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team.  Certain situations we can just deal with and not need to involve others.  If I need backup (additional resources), I can request those, but if a situation escalates, it’s time to bring in more support.  We don’t have a strict command and control environment;  as we grow our technical staff I think that’ll be more defined.

When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience.  Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role.  We currently don’t have anything defined like that, though I hope as we start to drill these scenarios more, I’ll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.

In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be.  By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.

In our case, I can envision that as having our PIO designate one chat room  or google hangout as our internal status update location and during the incident he/she’d make regular updates to those areas.  In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.

As for the Incident Commander role, we’d need to train all of our on-call personnel, as well  the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident.  While we don’t have this defined yet, we had a short outage a few weeks back that illustrates how this can work.

  • About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom.  I wasn’t the on-call person, but I always monitor for severe external alert failures.
  • I rolled out of bed and ran stumbled down to my office and got online to start investigating.
  • First order of business, check our chat site.. oops! It’s down!  Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it’s running in New York with the rest of our infrastructure.
  • Next up, VPN..  which connects.. that means internet to our data center in NY is still good.
  • I jump in a Google Hangout (fortunately unfortunately for my co-workers,  I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team.
  • Chat comes back online.  I dropped a note in chat that I’m in the SRE hangout and troubleshooting.
  • I’m soon joined by Geoff Dalgas, one of our Core Q&A developers.
  • He and I discuss the situation and began validating the different bit of our infrastructure.
  • We determined that we were seeing an issue with keepalived on our load balancers.
  • Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
  • We also had several more developers pop in and see if they could offer any help.
  • We determined a course of action to remediate the problem and began to implement it.
  • Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.

In this situation, I was first on-scene and acted as incident commander.  If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.

Open Line

So, what do you do in your organization?  Do you drill and train for failures?  Do you test your backups?  Do you prepare your operations personnel for how to respond in an incident?

I’ve been getting some great response to my previous post, and I wanted to make a few things very clear.

Clarifications

What I’m not doing -

  1. I’m not advocating or dismissing any particular configuration management tool.
  2. I’m not discounting the tough work done by companies and community projects that have created abstractions on managing disparate systems.

What I am attempting to do -

  1. Highlight the challenges of cross-platform management and application management.
  2. Show one of the efforts in providing a standards based management abstraction.
  3. Offer my thoughts on why I see value in that direction and what challenges I see.

Managing the Operating System vs. Managing Applications

There is definitely some confusion around using CIM to manage the OS vs managing applications.

  1. CIM Classes can be used to model applications as well as OS resources.
  2. Most of the “applications” that are packaged as roles and features in Windows Server expose a CIM management API.
  3. WMI is an implementation of the CIM standard and starting with Server 2012 and the Windows Management Framework V3, CIM is exposed via WSMAN.

Hyper-V, File Shares, Clustering, IIS, and others all offer CIM based management models.  Other applications can expose a CIM management model as well.  As long as the host CIM server (WMI on Windows and OMI or OpenPegasus or ???? on Linux based Operating Systems) is operational, applications can also offer their configuration and status via that channel by creating a provider.  To do this on Windows, there is some documentation to get you started:

To do this with OMI, you can find some documentation and source at https://collaboration.opengroup.org/omi/documents.php.

The Current State – Reprise

But, but, tool {fill in your favorite tooling here} already does THAT!!!

There are a number of tools that valiantly strive to provide cross-platform management.  I mentioned several of them in my last post, but there are a number of others.

Yep, it does.  Until…

things change.  The challenge these tools have is that they have had to implement their abstractions against very different implementations.  The problem there is that these things are not stagnant.  The management APIs can change over time and since there is not a standard description of the API or underlying configuration.

If CIM were the standard API exposing the configuration, the underlying implementation details can change, but configuration management and monitoring tooling don’t have to care about that.  The tool vendors and community projects can focus on other value adds for their particular tooling, rather than being forced to continually update the basics.

The Next Steps

We are still in the early stages of the push for CIM and WSMAN.  We’ll have to see how adoption picks up.  The continuing work around OMI holds promise, but it needs a deployment or integration story for various Linux distros and more public providers for managing components of the Linux OS and attendant applications.

There have been some interesting announcements at TechEd in relation to Windows Server 2012 R2 (watch this video and pay attention around 49 minutes in).  I’m going to talk more about this feature and it’s implications and my plans with it in the very near future.

 

​Configuration management today is mess if you work in a heterogeneous platform.

There is tooling that takes a stab at it, and is getting better (from the *nix world – Puppet, CFEngine, and Chef and from the Windows world – System Center Configuration Manager, Group Policy, among other third party application deployment platforms).  These tools are all well and good, but they fall down when reaching across the OS divide.  Puppet, Chef, and CFEngine (there are others as well, but these are some of the more popular) all have some cross platform support, but it feels unnatural (especially in module or recipe development).

Why is this a mess?

Windows is traditionally described as having an API oriented management model, whereas *nix has a document based management model.

Well, that’s a load of crappy, crap, crap.  What does that actually mean?

It means that the two operating systems offer two different management models.  The two different models have different abstractions and idioms for operating system constructs.  Let’s look at a concrete example, setting a static IP on a network interface (just the rough strokes.. I’m not going to spend too much time on the minutia). As I stated before, Linux uses a document oriented management model, so to configure my network interface, I’ll edit a document or two.

The Linux (Centos) example:

  1. Find the correct interface file under /etc/sysconfig/network-scripts
  2. Open it in your text editor of choice
  3. Edit it to contain your desired settings for the network interface and save the file
  4. If you need to add/modify DNS servers, find /etc/resolv.conf
  5. When done, you can bring your interface online with a command line call to
    ifup eth0

You’ll have something that looks like this for your network configuration file:

DEVICE=eth0
IPADDR=192.168.10.80
NETMASK=255.255.255.0
NETWORK=192.168.10.0
BROADCAST=192.168.10.255
GATEWAY=192.168.10.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
And something like this for your resolv.conf:
domain serverfault.com
search serverfault.com
nameserver 192.168.10.60
nameserver 192.168.10.61
That wasn’t so bad, and as an added benefit, they are just text files, so I could check them in to a revision control system (Versioning FTW!). Now, let’s look at what we’d need to do on the Windows side.  Since this is a blog for a community of “professional” systems administrators, we are going to dispense with any GUI example for doing this.

The Windows Server (2008 R2) example:

  1. Use WMI to retrieve the network adapter interface index.
  2. Use WMI to retrieve the network adapter configuration by the index.
  3. Set the desired IP address, gateway, and DNS servers and suffix against the WMI object.

You can use the following PowerShell commands to make those changes:

$NetworkAdapter = Get-WMIObject Win32_NetworkAdapter -filter "NetConnectionID = 'Local Area Connection'"
$NetworkAdapterConfiguration = Get-WMIObject Win32_NetworkAdapterConfiguration -filter "InterfaceIndex = $($NetworkAdapter.InterfaceIndex)"
$NetworkAdapterConfiguration.EnableStatic('192.168.10.80','255.255.255.0')
$NetworkAdapterConfiguration.SetGateways('192.168.10.1','1')
$NetworkAdapterConfiguration.SetDNSServerSearchOrder('192.168.10.60','192.168.10.61')
$NetworkAdapterConfiguration.SetDNSSuffixSearchOrder('192.168.10.60')
and

The Windows Server (2012) example:

  1. Set the desired IP address and gateway based on the interface name.
  2. Set the DNS servers and with a few more PowerShell commands.

You can use the following PowerShell commands to make those changes:

$IPAddressParameters = @{
            IPAddress = '192.168.10.80'
            InterfaceAlias = 'Local Area Connection'
            AddressFamily = 'IPv4'
            PrefixLength = 24
            DefaultGateway = '192.168.10.1'
}
Set-NetIPAddress @IPAddressParameters
Set-DNSClientServerAddress -InterfaceAlias 'Local Area Connection' -ServerAddresses '192.168.10.60','192.168.10.61'
Both of these examples are interactive commands, but I could easily save them in a file and place that under version control (and I should).

So what?

The examples don’t look all too different, but they do illustrate the difference between similar operations.  In both  examples, I end up with an artifact, but one is for a one time application of the setting (the Windows side) and the other is the setting storage location (the Centos example).

On the Centos box, we had to edit a file where the configuration was read from.  On the Windows servers, we updated settings via a WMI API (in both cases.. on Server 2012 there are more built in cmdlets, but many of them are thin wrappers over the WMI APIs) and not the actual end storage location.

This is what

Any configuration management tool that works in a cross platform capacity needs to understand these distinctions and check based on OS type which implementation to use when configuring a system.  This means for most configuration types, you’d have a big “IF” block where *nix based OS’s follow this line of processing and Windows based machines follow  the other line of processing.  This can become a maintenance nightmare as OS versions change the API on the Windows side or modify location and or layout of the configuration files on the *nix side.

And it’s even worse…

Now, what happens when you have a model that doesn’t translate across both worlds?

For example, how do manage file permissions?

Posix style permissions (used on most *nix variants) assigns permissions are nowhere as discreet as NTFS file permissions.  In addition on Windows, the file system auditing is also configured via the permissions configuration. In the reverse, on *nix files can be set as executable, where that is handled by file type mappings based on file extensions in Windows. This fragmentation leads to more complex implementations on the side of configuration management software developers or missing feature coverage.  In either case, this is a loss for the sysadmin who maintains cross platform environments.

But what if….

there was a common method of interacting with operating systems, regardless of what was running underneath? What if this method used a common transport (open standard) and communications were defined by an open standard? This is the direction Microsoft is taking with CIM and WS-Management.

CIM (Common Information Model) is a DMTF (Distributed Management Task Force) standard for describing management information for systems, networks, applications, and services.

WS-Management is another DMTF standard for management communication, focused on CIM traffic.

Microsoft has contributed to an open source project hosted by the Open Group called OMI.  OMI is a CIM server that communicates over WS-Management and is implemented to run on *nix based operating systems.

I’m personally interested in where this will go, given Microsoft’s market power (Cisco and Arista are working on incorporating OMI into their network switches).  The idea of a shared management model is very appealing to me, as I work in a cross platform environment.  I’m responsible for our Windows infrastructure, but I have to be able to work with our *nix infrastructure as well.  If I could use one model for interacting with both, that’s a huge win for me and my team.

This wouldn’t eliminate any domain specific knowledge on either OS side, as you’d still need to know what buttons to push and knobs to tweak to get things going and do some deep troubleshooting.  It does, however, make the idea standardizing how various OS components can be accessed, making basic configuration, monitoring, and troubleshooting much easier.

I’m interested because…

this pushes the implementation down to the OS provider (or the CIM provider provider) and gives vendors one target to hit for configuration standards.  In the Microsoft case, they can say “Follow this standard and any Windows system can manage you with minimal effort.”  If other OS’s support CIM and WS-MAN as well, it becomes easy to offer management interfaces there as well.

Obviously this would be huge change to the existing way of doing things for OS and application developers, not to mention systems administrators that are invested in their existing ways of doing things.

I don’t see another good alternative though, as the numbers and variety of systems continue to scale up and “cloud” becomes more of a factor in our environments, yet the number of admins is staying stagnant or being reduced.  Simplifying the management and monitoring surface makes sense in today’s and likely tomorrow’s data center landscape.

It doesn’t solve every problem and vendors can still implement vendor specific extensions (and we know how well that’s worked with SNMP).

 

NOTE – Be sure to check out some clarifications and expanded discussion in my followup post.

About a  year ago, I wrote a post about the basics of getting a job in Information Technology.  Unfortunately, that post (linked here) did not cover much relevant material when it comes to putting your resume in and getting an interview.  This blog post has been in the making for a long time, however the major impetus came when a serverfault regular tossed me a message over the weekend asking for suggestions on how to answer “The Salary Question.” I address this below, but first I will cover what I think is the single most important thing to remember in IT when it comes to getting hired.

“It’s not what you know, it’s who you know!”

I have heard many people bemoan the fact that it is near-impossible to get an interview for tech jobs in this market.  Their stories are generally the same, listed below:

  • See there is a position at Awesome Company, LLC.
  • See that the resume is asking for every technology buzzword imaginable
  • Asks for 10 years experience with Windows 8
  • Decide to send in your resume for it, even though you only know 3 or 4 of the 15 buzzwords they’ve specified.
  • Become depressed when you don’t get a callback.

This is not an unusual turn of events.  The reason behind this oftentimes is that HR is putting together a job description and they’re (optionally!) asking the team that’s hiring for “some technologies they should look for.”  Oftentimes that hiring team says “Well, if they know ABC, XYZ, TLA, ETLA, or Java, it’d be a plus.”   This turns into “applicant must have X years experience in ABC, XYZ, TLA, ETLA and Java.”  HR just screwed not only your chances of getting hired, but the hiring team’s chance at getting a potentially good candidate.

The thing that is tough for most people to understand is that all of the above is what I like to call an “HR Trap.”  It’s there to trap your resume and never get it to the hiring team for consideration.  So, as problem solving IT thinkers, we need to think around this problem.

Do you have any friends that work at Awesome Company, LLC?  Are they friends with the IT guys, per chance?  Ask your friends if they’d be willing to hand your resume to the team that’s doing the hiring along with a recommendation.  (NOTE: Don’t sound needy!  It’s a good way to sour friendships.)  Some people advocate saying “Hi James, I noticed your company is hiring for a position that I’d be great at, and I’d really love to get an interview.  Would you put my resume on Tim The IT Director’s desk for me and possibly put in a good word or two on my behalf?”  This can really work.

Network, Network, Network!

There’s another thing we can do to increase our chances of getting hired in the IT community: go meet people!  This is called “Networking” to the sales world and is critical in getting their job done.  I do know that a lot of us consider ourselves antisocial, but realistically, if all of us “antisocial” types get together, we generally find things to talk about.  Don’t be afraid to approach people, please!  This process only works when you are the one taking initiative.  Here are some options for networking:

  • Conferences (LISA, LOPSA-East (nee PICC), Cascadia, Velocity, many others) are great places to network with other people and rub shoulders with the giants of our industry.  For instance, Vint Cerf will be delivering the keynote at LISA this year.  If you ever wanted to meet him, if you were at LISA you could just go up and say hi.
  • Local user groups (Like LOPSA presentations, Linux User Groups, Windows User Groups, Tech Meetups) help a great deal as well, and for most of us without a big pocketbook, they’re much cheaper to attend than a conference.  You still get an opportunity to meet a gaggle of new people, and that’s helpful when you’re looking for a job.  I know that LOPSA especially, at least the chapter that I belong to, asks if anyone has job openings they’d like to talk about.  If someone’s asking for something you can do, go talk to them!  You’ve just instantly gotten ahead of the game by meeting the possible stakeholder(s) before HR has had the chance to accidentally delete your e-mailed resume.

Show the world how awesome you are.

An oft overlooked part of the hiring process is the inevitable google search that your potential employer is going to do.  What kind of stuff are they going to find?  Are your Facebook photos from your buddy’s beer pong tournament open to the public?  This is what could be called a “negative public artifact.”  Some tech companies would find it funny, but stuffier business types would consider that an immediate turn off.  On the other hand, if you have a blog full of insightful comments on the industry and diary entries of your (mis)adventures in technology, you show that you’re not simply a piece of paper with a lot of certifications, but someone who actually produces good things.  This is where being a member on a Stack Exchange site is very helpful.  If you mention you’re a serverfault user on your resume and provide the link to your profile, your potential employer has one place to look for all of your questions and answers to judge how well you know your stuff.

Fun With Resumes

I know a lot of us are holding onto the same resume format we had when we started, and are just adding fields as necessary; it’s not always a smart idea.  The job market and what people are looking for on resumes has changed since some of us started looking for jobs.  The emphasis is shifting from “show your experience” to “show how you’re awesome;” if you want your resume to get a second glance, it has to clearly and concisely illustrate why you’re a slam dunk.

1972-1977  Sanford and Son, Salvage

  • Drove a salvage truck
  • Excellent problem solver

If your resume has sections like the above, you are boring the heck out of the person reading your resume.  Sure, it shows what you literally did in that job, but it doesn’t convey what you’re proud of.  Did you make the salvage equivilent of the Kessel run in less than 12 parsecs?  How is the potential employer going to know about this if you didn’t say so?  Lets try this again:

Sanford and Son, Salvage (1972-1977)

  • Frequently noted as having the best one-liners in the company
  • Kept a 1952 Ford Pickup in running order much longer than previously considered humanly possible
  • Voted the fastest salvager on the east side after completing the Kessel Run in under 12 parsecs! (City record!)

This is not unusual advice.  I’m sure many of us learned it in school when they talked about writing resumes.  At the very least, we’ve spiced up an otherwise dull list of abilities/accomplishments.  Why stop there, we can still do better:

Sanford and Son, Salvage (1972-1977)

Accomplishments

  • Ran the Los Angeles Kessel Run in under 12 blocks, beating the standing record of 16 blocks. (1972)

Accolades

  • In Trash We Trust Award – 1973
  • Salvage Monthly’s “Top of the Heap” – 1975

Skills Utilized

  • Advanced engine mechanics and collision repair
  • Navigation of South-Central Los Angeles by automobile and commercial trucks
  • Lifting heavy loads

If we look at the above, two things are important to note.  First, it has easy to read sections.  Second, it has superfluous space taken up by sections.  If you were to put the whole body of your experience down in this format, you’d have a 4-5 page resume.  Some places might like this, but the common knowledge is 2-3 pages, max.  Without some massaging, your long-winded resume will be too troublesome to read.  Why?  Oftentimes we’re handed a stack of resumes and after a while, it all gets so repetitive that you confuse which applicant had which experience.  The key to resolving this is massaging your work experience to be relevant to the job that you’re applying for.

Tailor that resume, chief

I know that it’s hard work to tailor resumes for every position, but at the very least you should have a few tailored resumes in the areas that you are most proficient.  For example, I have three separate resumes that I give out if I’m applying for a developer, IT, or management position.  I’ve done all three jobs, so I have three tailored sets of job experience that subtly (or perhaps not-so-subtly) emphasize those roles in the full body of my experience.  Sometimes I leave a job out that isn’t relevant to the position, but I’ve found some interviewers are taken aback when you mention you left out some job experience.  Why would  you care if I slung lattes at a coffee bar while in high school, I ask?

Cover letters are mandatory

You might have heard people say that cover letters are dying out in this age of digital resumes.  They’re lying so they can take your jobs.  Every single resume you sent out needs a cover letter, and it needs to be customized to the company you want to work for.  I know that this is a tall order for some, but if you’re a prudent applicant you’ve already studied the company you’re applying for and know what they do or provide.  Why not explain to the person who’s going to hire you why they should waste their day interviewing you?  The cover letter is your “First Impression” on paper.  It helps to show how eloquently you convey yourself on paper, and it’s  pretty important in tehnology jobs because a lot of our jobs involve some level of technical writing.  Ignore this advice at your peril.

Please provide your starting salary…

Yeah, if you want to get priced out of the market.  The answer to that question should always be “we can negotiate that after the interview when you’re sure I’ll be a great fit!”  If they press, try to dodge as much as possible.  The salary question allows them to automatically throw out resumes.  If you come in low, they think you’re not in touch with the job requirements or the job market.  If you come in high, they’ll think you’re either presumptuous or literally are too experienced and therefore too expensive to hire.

Your resume will give them an accurate idea of how much they should be paying you just by the virtue of how much experience you’re showing.  If you’d like more details on this, check out this year’s Robert Half Salary Guide.  This gives an accurate starting point to figure out how much money you should be asking for or how much they should be paying you.  However, don’t tell them this number (or your imaginary What-Am-I-Worth-A-Year number) until after they’ve decided they want to hire you.

Closing thoughts

I don’t want to end this on a somber note, but please keep in mind that if you’re trying to get interviews at companies where you have no inroads or networking contacts, please be ready to get a callback from 1 out of every 20 or more resumes you send.  It is a cut-throat job market and there are hundreds if not thousands of people applying for the same positions you are, but they’re better at bullshitting on resumes than you are.  It’s tough to get a job without having friends in the industry already.  Don’t get discouraged!  Do what I advise above, especially on the socializing front, and you’ll be much better off.

Appendix 1: What a well-formed Systems Administrator job reference might look like

I was asked to provide an actual SysAdmin-focused job description to help some people along.  Below is a completely fictitious entry that I’ve generated off the top of my head.

Senior Systems Administrator, ImaginedCompany, LLC (2008-Present)

Relevant Technologies

  • Advanced Active Directory provisioning and maintenance for a 100,000 user company
  • Deployment automation via Windows Deployment Services in Windows and KickStart in Linux
  • vSphere setup, troubleshooting and administration with multiple Datacenters and Hosts
  • Deep Telephony – PRI, TDM, FXO/FXS setup and maintenance
  • Configuration Automation via Puppet and Systems Center
  • Revision control with Mercurial, Git and Subversion

Accomplishments

  • Independently developed the Windows Deployment Services installation for ImaginedCompany, decreasing provisioning time from hours to minutes.
  • Converted legacy servers from bare-metal hardware to vmware, saving the company thousands of dollars a year in electricity
  • Quickly reimplemented a PBX solution when the legacy telephone system hardware failed, enabling the company to continue to make money until a permanent solution was in place.
  • Led-by-example with new project to use source and configuration management by way of Mercurial and Puppet/Systems Center

Accolades

  • 2008 – LOPSA Most Active Mentor
  • 2009 – Golden Baseball Bat recipient for best client-facing customer service.
  • 2011 – Employee of the Year

It is pretty hard to get a bunch of system administrators together for any period of time and not have a conversation about command line for managing things. In my experience, command line always wins the debate for any medium to large installation. Microsoft caught on to this and has introduced powershell. From my own experience and people I have talked to, powershell adoption has been slow. Like powershell or dislike it, it is an interesting take on command line with its object oriented versus text-based approach. It is also quite powerful, so why the slow adoption?

In my mind it isn’t a problem with powershell itself, but rather a result of human nature, culture, and the Windows ecosystem. The problem lies in workflow, and I think the best way to illustrate this is to contrast Windows versus Linux administration. When managing Linux systems, scripting something flows naturally from the experience of trying things out in the first place. For example, lets say someone with some Linux experience (but not an expert) is building a piece of software and installing it on Linux for the very first time. The steps usually involve something like:

  1. Run the traditional, configure, make, make install from the command line. See what libraries or binaries are missing that are required to build, from the errors. Install them (often via yum or apt), keep doing this until it works.
  2. Possibly add some users to the system, adjust permissions etc with commands like useradd, chown, chmod.
  3. Edit some text files
  4. Set it to start at boot with commands like chkconfig or update-rc.d

As you do this more and more, you learn the patterns of what fails, and get the urge to automate it. Although things like puppet are the end goal these days, the next normal step is to write shell scripts to do this. This is the lynchpin of the whole experience, a shell script is just a little flow control, error handling, and a few variables on top of what you were already doing. With the Linux experience, scripting flows naturally as a next step in your experience. It is a gradual learning experience, and your previous experience of doing things manually taught you much of what you need to know to script it — what goes wrong, what to check, how to do everything you need to do. Even the first time tutorials on the web are almost always command line. From the second you start learning how to administrate a Linux box, you are learning how to script.

Unless you are extremely disciplined or have been taught Windows administration recently by a “benevolent” dictator, your first steps in Windows administration are via the GUI and Wizards. Don’t get me wrong, for many things this gets the job done faster and with far more ease. The generally accepted problem with the GUI though is that it doesn’t scale. When you want to scale in management you often turn to scripts (GPOs are very useful, but you are generally out of luck if what you want isn’t predefined.) With Windows, scripting these tasks is generally a complete departure from your previous experience. Scripting doesn’t flow naturally from previous experience, and scripting something is scary because of this. This means the windows ecosystem, due to human nature, is far less likely to produce administrators with scripting experience. Therefore, powershell adoption is logically slower. This also means administrators are less likely to be testing powershell and pressuring each other and Microsoft to improve it.

The cure to this workflow to me seems to be Windows Core since changing human nature isn’t easy. With Windows core, even though you can use the Management Console, you really have to start doing many things via the command line and powershell. With use of these tools, just as with the shell in Linux, scripting will flow naturally. The Catch-22 however though is that since adoption is slow, everything you need might not be available via command line options. At my company we haven’t used Windows core, but I’m hoping with Server 2012 it is time for Core, does anyone with more experience know if it is time for this shift?

OODA for Sysadmins

Kyle Brandt

Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.

OODA in Practice

As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.

Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.

Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.

The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.

This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.

Common Problems

This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.

When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.

Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.

If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.

OODA Done Right

Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.

With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.

Why it Matters

If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.

I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries.  Being a sysadmin really requires a love of computers.  For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked.  How amazing and wondrous this world could be!  So many cool toys!  “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.

One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects.  I, for one, have several projects open on my workbench in the basement that are in varying states of completion.  I can get away with this because they’re personal endeavors and the only person I’m accountable to is me.  The problem comes when we translate this type of behavior into the world of business.

Your Office is Not Your Basement

The business world exists really for one purpose:  turning a profit.  Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost.  Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.”  All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.

Lets face it: we love working on fun computer projects.  However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done.  They’re what I like to call the “toilet plunging” of being a data janitor.  Nobody wants to do it, but it has to be done.  I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.”  Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.

Being Thorough is a Learned Habit

We don’t come out of the womb with an innate ability to understand all of the implications of our choices.  This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives.  Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill.  Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.

Everyone will eventually learn this lesson the hard way.  At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself.  Did you plan how you were going to execute the project?  Did you consider any kind of pitfalls?  Did you do any research, if necessary, beforehand?  When executing the project, were you sure to do it in a way that would be easy to maintain in the future?  Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you?  These are all things we need to be aware of when we’re doing our jobs.

It Might be a Reason, But Not An Excuse

It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job.  There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct.  A big one I hear is “this is only going to be temporary.”  It won’t be, trust me.  You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline.  Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”

Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.”  Every time I hear this I want to strangle someone!  First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south?  Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned.  Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do.  If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk.  If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated.  This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.

On Feeling Overwhelmed and Slog Overflow

There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning.  There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain.  Once this happens, you’re in “Slog Overflow.”  This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms.  It is the point where procrastination begins to win out over doing a job you once loved.  At some point, you wake up and realize you don’t want to go to work anymore.  You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk.  To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.”  You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying.  If you feel like your world is moving in this direction, TAKE ACTION NOW!

Here are some thoughts on how to fix these problems or prevent them from happening in the future:

  • Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do.  Be sure that it includes pre- and post-task steps so that the whole process is considered.  If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
  • Embrace feedback and criticisms – We’re all in this together!  While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them.  Take what is useful, try not to be offended.  Nobody’s perfect!
  • Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time.  That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation.  Demand a higher level of awesome from the people you work with, too.  It only takes one hole for the boat to sink.
  • Teach young Systems Administrators to be thorough – Lead by example.  Junior sysadmins absolutely pick up behaviors from senior team members.  If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.

What are your experiences with these problems?  Feel free to comment below; I love feedback.

If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

  • More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
  • Licensing Costs
  • Owned vs Rented Assets
  • How many cloud “servers” or instances you would need vs real hardware
  • Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.