In order for system administrators to do their job well, particularly in a tech company, they need to know a lot of what is going on. This is because just about everything is done on the systems we control.

Lets look at some of examples of things system administrators probably need to know, and why they need to know them:

Example 1: Upcoming projects

In order to make sure we have enough capacity in servers, network, backups, etc we need to know what is incoming. If we don’t, it can be a lot more difficult to be prepared and that can slow things down.

Example 2: How a service or code works

System administrators are generally the first line of troubleshooting. In order to troubleshoot a problem, we need to know what is being done before we can trying to figure out why it isn’t working. We also need to monitor and backup the system, knowing how it works tells us what details to monitor and what data needs to be backed up.

Example 3: How important something is to the company

Resources are always limited. Although you want minimum standards of things like monitoring and backing up, time and money is limited — system administrators need some context for setting priorities. This can also help with figuring out an appropriate level of security.

Example 4: What people do

System administrators control access, so we need to have an idea of what sort of access people should have. We also need to know the best people to talk to when their is a problem or there is maintenance to do.

Knowing without Being Nosey

If we accept that system administrators really do need to know quite a bit of what is going on, then system administrators need to figure out how to do this without being nosey:

Definition of NOSY
: of prying or inquisitive disposition or quality : intrusive

The challenge is to have a good handle on what is going on, without prying or being intrusive. Part of the difficulty is that this is a two step process:

  1. Find a way to sincerely not be nosey
  2. Don’t come off as being nosey

These two steps are not easy, and require constant vigilance. If you have mastered them, then you probably don’t have to ask for information most of the time — information will be given to you and you will be invited to be part of the process.

Getting to that point is tricky, and I certainly don’t claim to have all the answers. In part it requires the cooperation of the other people in your company, but if we hold up our end of the bargain it goes a long way.

So what can system administrators do?

  1. Don’t be nosey. Make it clear that knowing this information is not for your entertainment or to make you feel special, rather it is to enable you to better do your job.

  2. Make things easier. Although sometimes doing your job requires you to get in the way, you should strive to add requirements because it makes things for everyone easier in the long run, not to exert power or justify your existence. If you don’t need to actually add a process or make things more difficult — then don’t. In many companies you want to be conservative with how much process you add.

  3. Be consistent. Telling one person on the system administration team should be as good as telling everyone. Once you get involved, document, backup, and monitor everything. If your team is consistent it goes towards developing a reputation of making things easier for everyone.

  4. Be respectful. If you work with great people, making sure things are good on the system side should be about being thorough. It is an SA’s job to think about that side of things full time, but it doesn’t mean the people you work with didn’t already think about it, or are being dumb if they didn’t.

  5. Know your place. If you are invited into the process of a new project, keep in mind why you are there. If you have a really good idea out of your area of expertise try to share it tactfully. But if you are there mostly to listen, then try to mostly just listen.

In the end I think knowing everything that is going on, without being nosey, is pretty difficult. Most of us at some time or another have probably failed at some of the things I listed — it takes some honest self evaluation to find where you are falling short. Any readers have ideas for how to stay apprised of everything without being nosey?

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

  • More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
  • Licensing Costs
  • Owned vs Rented Assets
  • How many cloud “servers” or instances you would need vs real hardware
  • Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

On more than one occasion I have been asked “how do I get a job in IT?” This question could easily be shrugged off and relegated to the canned answers pile; phrases I have personally uttered in the past include “go get some certifications then send out resumes” or “play with computers for a few years and then apply.” These answers really are a cop-out on my part so I’d like to take a few minutes to apply some serious thought to the query and offer my thoughts.

What People Will Expect of You

While most people would say that having a large breadth and depth of knowledge in computers is the primary requirement for being employed in IT, I would disagree with this. There are a handful of very critical attributes that you will find in any IT Rockstar:

  • Good Problem Solving Skills
  • Critical Thinking
  • Strong work ethic
  • Ability to handle “Burst Stress”

Problem Solving is possibly the most important attribute any good IT team member will have. Problem solving is the core of our vocation, when you think about it. IT is tasked with solving problems that the other employees aren’t able to handle. You’ll also find that in many workplaces, the IT guys are the ones helping to solve other complex problems in the enterprise because of the problem solving skillset they demonstrate. Critical Thinking is another important skill to have when talking about technical jobs. Oftentimes the problem I’m trying to solve does not have an easy-to-see solution (sometimes, the cause itself is not immediately apparent.)

Do you like working long hours for little praise or thanks? That’s pretty much working IT in a nutshell. Performing server maintenance usually needs to wait until after hours, which generally means you’ll be putting in a full 8 hours for that workday, then many more hours that night to do the after-hours downtimes. Couple this with “burst stress” and you’ve got a recipe for gray hair. Burst Stress is common in jobs like police work or firefighting, where you have long periods of low stress followed by short bursts of extremely high stress situations.

What do you want to do?

There are a handful of disciplines in IT, each with their separate purposes. What follows is a breakdown of the individual disciplines and what that type of work entails. Regardless of what path you choose, you’ll likely start in an IT Generalist role, and many people stay in that role their entire career. I myself genuinely enjoy being a generalist; it means that any given day I might be working on any number of problems that aren’t the same old issues over and over again.

IT Generalist — This is often the least respected but most useful role in IT. As a generalist, you are a Jack of All Trades. You’re expected to understand not only Desktop and Server Support, but also have a useable knowledge set in Telephony, Networking, Backup and Disaster Recovery, and Security. When you start your new IT career, you’ll usually start as a quasi-apprentice in a generalist role. Once you’ve “earned your stripes” in a few disciplines, you’ll be more educated and ready to move to a specialization, if that’s your pleasure.

Desktop Support — The Desktop Support team will generally be tasked with maintaining the software and equipment that other people in the company utilize to get their jobs done. This job is very demanding from a customer service perspective, as the person doing this job will oftentimes be assisting users who are already annoyed that their machines aren’t working the way they’d like! Skillset wise, Desktop engineers need to be highly fluent in whichever desktop operating system your company uses, as well as any applications the company makes use of. Generally speaking, you’re going to need to know Windows and Office like the back of your hand if you want to get a job like this in the majority of businesses.

Server Support/Systems — The Systems or “Server Support” team has similar requirements as Desktop Support when it comes to strong knowledge sets of operating systems and software. The jobs diverge when you look at the “back-of-the-house” operations that a Systems team is often assigned. Knowledge of datacenter operations (power distribution, hot row/cold row, rack positioning) is essential in this role if your organization has more than just a “closet in the back with a few servers.” Server Support is also usually in charge of all backend/utility systems in the organization, including directory authentication, mail and groupware, and administrating backups.

Networking — This group focuses mostly on interconnecting sites and equipment together. To get into this group, you better be prepared to prove you know the OSI model like the back of your hand. This is a tough group to coast in; TCP/IP can be fickle under wavering hands. Usually, a particular company will standardize on one vendor for their equipment, and you’ll be expected to know the operating system for that particular vendor. If you’re looking to get into the networking team at a new company, you should likely know Cisco and Juniper OS’s (Cisco IOS/ASA-OS, JunOS, etc.) Even if the company does not use either of these major brands, having knowledge of how other platforms operate can assist you in doing the job. Networking also is susceptible to the above mentioned “Burst Stress”, since if a site-to-site link is down, it’s possible that hundreds of people are sitting waiting for you to fix the problem.

Security — To be a member of the security team is to be alternately loved and hated by differing groups in your organization, often different groups at different times. Security generally finds themselves tasked with doing audits of internal data security and setting standards to help achieve compliance with data security standards. Computer forensic applications are the toolbox of the IT Security team; to do the job it will help to be seasoned at data analysis/data mining and event correlation. Security is usually the team tasked with defending audits as well, so it helps if you have good verbal communication skills. Finally, Security is sometimes tasked with coming up with the Disaster Recovery and Business Continuity Plans for a business. In some cases, other departments handle this, but in my experience this has been a Security-type role.

Database Administration — DBA is a tough specialization to be hired in. Brent Ozar has a great blog post covering this, so I won’t go in to too much detail but instead refer you to Brent Ozar’s excellent blog post series linked here. Suffice to say, you need to be able to speak SQL and make it sound like Shakespeare before you can roll with the titans like Mr. Ozar.

Telephony — I’m hesitant to group Telephony into IT, but many times it falls under the moniker of IT, so we’ll cover it here. Telephony is the specialization that handles telephones and telephony technologies. You’re going to want to know the details of how signaling works in PSTN networks, things like TDM, FXO,FXS, E&M, CAS, and a whole lot of other crazy acronyms. Depending on the size of the organization, you might also need to have some cross-pollination with the Networking skillset.

In a future blog post, I will cover what I consider the “elephant in the corner” when it comes to hiring in IT. Remember, it’s not always what you know, but who you know that gets you hired. As always, your comments and criticisms are welcome/encouraged; I’d like to hear what the community feels on this topic.

Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.

At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.

Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?

Why Use Virtualization?

Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.

This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:

  • Live migration (vMotion in VMWare terms)
  • Running multiple operating systems (i.e. Windows and Linux) on the same hardware
  • Easier to get full utilization of hardware resources by moving VMs around

These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?

The Joy of Commodity Hardware

As Wikipedia defines it:

Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”

Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.

So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.

Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.

Virtualization and Centralized Storage are Best Friends

With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.

Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.

Virtualized Clusters to the Rescue?

If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.

VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.

Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.

What Will We Go With?

Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.