In order for system administrators to do their job well, particularly in a tech company, they need to know a lot of what is going on. This is because just about everything is done on the systems we control.
Lets look at some of examples of things system administrators probably need to know, and why they need to know them:
Example 1: Upcoming projects
In order to make sure we have enough capacity in servers, network, backups, etc we need to know what is incoming. If we don’t, it can be a lot more difficult to be prepared and that can slow things down.
Example 2: How a service or code works
System administrators are generally the first line of troubleshooting. In order to troubleshoot a problem, we need to know what is being done before we can trying to figure out why it isn’t working. We also need to monitor and backup the system, knowing how it works tells us what details to monitor and what data needs to be backed up.
Example 3: How important something is to the company
Resources are always limited. Although you want minimum standards of things like monitoring and backing up, time and money is limited — system administrators need some context for setting priorities. This can also help with figuring out an appropriate level of security.
Example 4: What people do
System administrators control access, so we need to have an idea of what sort of access people should have. We also need to know the best people to talk to when their is a problem or there is maintenance to do.
Knowing without Being Nosey
If we accept that system administrators really do need to know quite a bit of what is going on, then system administrators need to figure out how to do this without being nosey:
Definition of NOSY
: of prying or inquisitive disposition or quality : intrusive
The challenge is to have a good handle on what is going on, without prying or being intrusive. Part of the difficulty is that this is a two step process:
- Find a way to sincerely not be nosey
- Don’t come off as being nosey
These two steps are not easy, and require constant vigilance. If you have mastered them, then you probably don’t have to ask for information most of the time — information will be given to you and you will be invited to be part of the process.
Getting to that point is tricky, and I certainly don’t claim to have all the answers. In part it requires the cooperation of the other people in your company, but if we hold up our end of the bargain it goes a long way.
So what can system administrators do?
Don’t be nosey. Make it clear that knowing this information is not for your entertainment or to make you feel special, rather it is to enable you to better do your job.
Make things easier. Although sometimes doing your job requires you to get in the way, you should strive to add requirements because it makes things for everyone easier in the long run, not to exert power or justify your existence. If you don’t need to actually add a process or make things more difficult — then don’t. In many companies you want to be conservative with how much process you add.
Be consistent. Telling one person on the system administration team should be as good as telling everyone. Once you get involved, document, backup, and monitor everything. If your team is consistent it goes towards developing a reputation of making things easier for everyone.
Be respectful. If you work with great people, making sure things are good on the system side should be about being thorough. It is an SA’s job to think about that side of things full time, but it doesn’t mean the people you work with didn’t already think about it, or are being dumb if they didn’t.
Know your place. If you are invited into the process of a new project, keep in mind why you are there. If you have a really good idea out of your area of expertise try to share it tactfully. But if you are there mostly to listen, then try to mostly just listen.
In the end I think knowing everything that is going on, without being nosey, is pretty difficult. Most of us at some time or another have probably failed at some of the things I listed — it takes some honest self evaluation to find where you are falling short. Any readers have ideas for how to stay apprised of everything without being nosey?
Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.
Cloud vs Self Hosting Cost?
The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:
- More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
- Licensing Costs
- Owned vs Rented Assets
- How many cloud “servers” or instances you would need vs real hardware
- Cost differences when you consider high availability
To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.
We Love Computers
and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.
This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.
We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.
On more than one occasion I have been asked “how do I get a job in IT?” This question could easily be shrugged off and relegated to the canned answers pile; phrases I have personally uttered in the past include “go get some certifications then send out resumes” or “play with computers for a few years and then apply.” These answers really are a cop-out on my part so I’d like to take a few minutes to apply some serious thought to the query and offer my thoughts.
What People Will Expect of You
While most people would say that having a large breadth and depth of knowledge in computers is the primary requirement for being employed in IT, I would disagree with this. There are a handful of very critical attributes that you will find in any IT Rockstar:
- Good Problem Solving Skills
- Critical Thinking
- Strong work ethic
- Ability to handle “Burst Stress”
Problem Solving is possibly the most important attribute any good IT team member will have. Problem solving is the core of our vocation, when you think about it. IT is tasked with solving problems that the other employees aren’t able to handle. You’ll also find that in many workplaces, the IT guys are the ones helping to solve other complex problems in the enterprise because of the problem solving skillset they demonstrate. Critical Thinking is another important skill to have when talking about technical jobs. Oftentimes the problem I’m trying to solve does not have an easy-to-see solution (sometimes, the cause itself is not immediately apparent.)
Do you like working long hours for little praise or thanks? That’s pretty much working IT in a nutshell. Performing server maintenance usually needs to wait until after hours, which generally means you’ll be putting in a full 8 hours for that workday, then many more hours that night to do the after-hours downtimes. Couple this with “burst stress” and you’ve got a recipe for gray hair. Burst Stress is common in jobs like police work or firefighting, where you have long periods of low stress followed by short bursts of extremely high stress situations.
What do you want to do?
There are a handful of disciplines in IT, each with their separate purposes. What follows is a breakdown of the individual disciplines and what that type of work entails. Regardless of what path you choose, you’ll likely start in an IT Generalist role, and many people stay in that role their entire career. I myself genuinely enjoy being a generalist; it means that any given day I might be working on any number of problems that aren’t the same old issues over and over again.
IT Generalist — This is often the least respected but most useful role in IT. As a generalist, you are a Jack of All Trades. You’re expected to understand not only Desktop and Server Support, but also have a useable knowledge set in Telephony, Networking, Backup and Disaster Recovery, and Security. When you start your new IT career, you’ll usually start as a quasi-apprentice in a generalist role. Once you’ve “earned your stripes” in a few disciplines, you’ll be more educated and ready to move to a specialization, if that’s your pleasure.
Desktop Support — The Desktop Support team will generally be tasked with maintaining the software and equipment that other people in the company utilize to get their jobs done. This job is very demanding from a customer service perspective, as the person doing this job will oftentimes be assisting users who are already annoyed that their machines aren’t working the way they’d like! Skillset wise, Desktop engineers need to be highly fluent in whichever desktop operating system your company uses, as well as any applications the company makes use of. Generally speaking, you’re going to need to know Windows and Office like the back of your hand if you want to get a job like this in the majority of businesses.
Server Support/Systems — The Systems or “Server Support” team has similar requirements as Desktop Support when it comes to strong knowledge sets of operating systems and software. The jobs diverge when you look at the “back-of-the-house” operations that a Systems team is often assigned. Knowledge of datacenter operations (power distribution, hot row/cold row, rack positioning) is essential in this role if your organization has more than just a “closet in the back with a few servers.” Server Support is also usually in charge of all backend/utility systems in the organization, including directory authentication, mail and groupware, and administrating backups.
Networking — This group focuses mostly on interconnecting sites and equipment together. To get into this group, you better be prepared to prove you know the OSI model like the back of your hand. This is a tough group to coast in; TCP/IP can be fickle under wavering hands. Usually, a particular company will standardize on one vendor for their equipment, and you’ll be expected to know the operating system for that particular vendor. If you’re looking to get into the networking team at a new company, you should likely know Cisco and Juniper OS’s (Cisco IOS/ASA-OS, JunOS, etc.) Even if the company does not use either of these major brands, having knowledge of how other platforms operate can assist you in doing the job. Networking also is susceptible to the above mentioned “Burst Stress”, since if a site-to-site link is down, it’s possible that hundreds of people are sitting waiting for you to fix the problem.
Security — To be a member of the security team is to be alternately loved and hated by differing groups in your organization, often different groups at different times. Security generally finds themselves tasked with doing audits of internal data security and setting standards to help achieve compliance with data security standards. Computer forensic applications are the toolbox of the IT Security team; to do the job it will help to be seasoned at data analysis/data mining and event correlation. Security is usually the team tasked with defending audits as well, so it helps if you have good verbal communication skills. Finally, Security is sometimes tasked with coming up with the Disaster Recovery and Business Continuity Plans for a business. In some cases, other departments handle this, but in my experience this has been a Security-type role.
Database Administration — DBA is a tough specialization to be hired in. Brent Ozar has a great blog post covering this, so I won’t go in to too much detail but instead refer you to Brent Ozar’s excellent blog post series linked here. Suffice to say, you need to be able to speak SQL and make it sound like Shakespeare before you can roll with the titans like Mr. Ozar.
Telephony — I’m hesitant to group Telephony into IT, but many times it falls under the moniker of IT, so we’ll cover it here. Telephony is the specialization that handles telephones and telephony technologies. You’re going to want to know the details of how signaling works in PSTN networks, things like TDM, FXO,FXS, E&M, CAS, and a whole lot of other crazy acronyms. Depending on the size of the organization, you might also need to have some cross-pollination with the Networking skillset.
In a future blog post, I will cover what I consider the “elephant in the corner” when it comes to hiring in IT. Remember, it’s not always what you know, but who you know that gets you hired. As always, your comments and criticisms are welcome/encouraged; I’d like to hear what the community feels on this topic.
Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.
At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.
There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.
Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.
Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.
Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.
We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.
At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.
Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?
Why Use Virtualization?
Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.
This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:
- Live migration (vMotion in VMWare terms)
- Running multiple operating systems (i.e. Windows and Linux) on the same hardware
- Easier to get full utilization of hardware resources by moving VMs around
These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?
The Joy of Commodity Hardware
As Wikipedia defines it:
“Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”
Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.
So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.
Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.
Virtualization and Centralized Storage are Best Friends
With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.
Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.
Virtualized Clusters to the Rescue?
If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.
VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.
Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.
What Will We Go With?
Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.
One website defines production control as the following:
Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.
That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.
Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.
Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.
As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.
An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.
For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.
Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”
The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.
Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?
We have gone and done it, we’ve expanded our Systems Administration team to THREE people. I’d like to introduce you to Peter Grace who we have just brought on board as our third sysadmin.
This is the second company I’ve had the honor of working with Pete at, and I can say that he is an exemplary System Administrator. In addition to being a great sysadmin, Pete is a fellow gun enthusiast, which increases our zombie attack survival rate by 35%.
Pete and I have done many great things together in the past, and I’m very excited about what he brings to us and the great things he, Kyle and myself are going to do as we move forward doing our part to make the internet better.
I have always had the notion that some companies “Get It” when marketing to programmers, sysadmins, and hackers. A couple weeks ago I was in Silicon Valley at Tech Field Day listening to a lot of presentations with some fellow geeks. In some of these presentations they “got it”, in others they didn’t.
The fundamental difference is that the ones that get it tell me about their technology, and the ones that don’t tell me about their products. True geeks have one common trait, and that is that they like to learn how things work. We get excited about learning and news ideas. Exciting technologies solve problems in non-obvious ways. Since these are not obvious, we want to learn about them, and in as much detail as possible.
Products, on the other hand, are how these ideas are packaged. For the most part, we don’t care. If we do care, that comes later, only if it will solve a problem that we have will we start to care about implementation and packaging. The advantage of telling us about your technology is that we do care, even if we don’t have a use for it at the moment, because we are geeks – it is our nature. When done right, we will then associate your product with the technology and that will be enough.
This has some consequences that marketing should be aware of when targeting tech people. First, we generally don’t want to talk to you directly, at least, not for very long. This is not because you are not important or interesting, it is just that you probably can’t get us excited about your technology like your engineers can. You can enable your engineers to present your technology well, and that is what people who are good at geek marketing do. If you can’t get your actual engineers to present no mater how hard you try, your screwed. It’s not your fault, your company just sucks and you probably just need to move on. Lastly, everything targeted to tech people should be aimed at getting us excited about your technology, not your product.
With my recent experiences at tech field day there were some good examples of this done right. Pure Storage taught me about why I see SSDs fail and a new type of RAID they invented suited for SSDs. Arkeia educated me about various implementations of deduplication in backups. Data Direct Networks introduced me to the concept of object store filesystems. However, in that case I wanted to learn even more given that amount of time. How successful they were came down to how much time they all devoted to fulfilling my desire to learn more about technology vs. telling me about their products.
I’ve recently been looking back on what we have written about our architecture in the past, and came to a stunning realization. That realization is that while we have many many different articles about what we have been doing there hasn’t been a good, solid overview of our architecture in a long time. In fact, the last really comprehensive write-up was done by Jeff before this blog even existed. And, boy I do have to say there has been quite a lot of change behind the scenes since then. So, my dear readers I’m going to take some time – and my next few blog posts – to give everyone an in depth look into how we have the Stack Exchange Network setup to serve between 12 and 14 Million page views per day.
How these posts will breakdown
Since we have obviously grown, and are offering more services to our users I’m going to break these posts out by each of the 4 major services we offer to our user base:
- Core Q&A (this includes the API)
- Community Blogs
Each one of these systems all work towards our goal of making the internet better, but they have different requirements and different challenges.
In this first post, I’ll be focusing on our core Q&A system, since that is after all our bread and butter.
First, a high level overview of how everything is put together:
Our core hardware setup hasn’t changed all that much. Well, I should say the chassis haven’t changed that much. We’ve done a lot of work to upgrade the internals of the servers when needed to address performance issues as they came up, as well as handle issues that resulted from Stack Overflow being so big.
Of these 10 Servers, 3 are dedicated to Stack Overflow with an additional 3 servers serving Stack Overflow and the Stack Exchange Network. We have one server dedicated to Dev/QA – which also hosts meta.stackoverflow.com. Our Web Tier machines normally operate between 5 and 20% utilization. We have plenty of room to grow on these boxes.
- 10 Dell R610 IIS web servers:
- 2x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
- 16 GB RAM
- Windows Server 2008 R2
- 2 drives
- RAID 1
- 2x Intel 320 300GB SSD (RAID 1)
We have two database server pairs. One pair is dedicated to running Stack Overflow, and the other runs the rest of the network. We run development against the secondary server of the non-stack overflow database pair. Both of our database pairs run at about 20% utilization, so once again we have room to grow here as well.
- 2 Dell R710 database servers:
- 2x Intel Xeon Processor X5680 @ 3.33 GHz
- 96 GB RAM
- 8 spindles
- Mirrored Pair for OS
- 6 disk RAID10 for databases
- SQL Server 2008 R2 SP1
- 2 Dell R710 database servers (Stack Overflow Dedicated):
- 2x Intel Xeon Processor X5680 @ 3.33 GHz
- 96 GB RAM
- 8 drives
- Mirrored Pair for OS
- 6 drive RAID10 of Intel X25-E SSDs for Database
- SQL Server 2008 R2 SP1
We run redundant Redis servers for our caching tier.
- 2 Dell R610 Redis servers:
- 2x Intel Xeon Processor E5640 @ 2.66 GHz
- 16 GB RAM
We use HAProxy for our load balancing, and Cisco Switching.
- 2 Dell R610 HAProxy servers:
- 1x Intel Xeon Processor E5640 @ 2.66 GHz
- 4 GB RAM
- Ubuntu Server
- 6 WS-C2960S-48TS-L Gigabit switches
- FlexStack (two stacks, 4 switches and 2 switches)
As with any system, making sure that your data is backed up and the backups are good is an integral part to your service offering. We backup our databases nightly and restore them to two different locations. One local to our NY data center for our devs to work against, and one remote in our OR data center.
Overall I believe that we are in a good place and have plenty of room to grow given our current setup. As always we will constantly be looking at our infrastructure and tweaking it to get the best performance possible and give our users the best experience possible.
System administrators can sometimes seem like they are trying to get everyone down. Someone goes to them with a great idea, application, feature, or just something they just want to get done. When the sysadmin comes in, instead of being enthusiastic about the great idea, he starts talking about viruses, hacking, earthquakes, floods, fires, Vogon invasions, and all the things that can go wrong. Because of this, the natural reaction is to stick the sysadmin in the basement and stop inviting them to parties.
In order to work past this issue, it takes a little work from both sides of the table. Coworkers need to remember that disaster is part of a sysadmin’s job and that Murphy’s law is part of what they do. Part of a sysadmin’s job to think about this stuff all the time because we are expected to have backups and to be able to recover from small events to major disasters. Major disasters are uncommon but do happen, hence the importance of offsite backups at a minimum. Small disasters, such as power failures, router failures are actually quite common. When a sysadmin is doing a really good job, people don’t even know about some of the smaller failures.
It also falls on the sysadmin’s shoulders not get stuck in this gloom and doom world. We have remind ourselves not to get in the habit of saying No to everything because something could go wrong. This is important because the danger is that that people will try to start bypassing you or just not believing you when you talk about very real threats because they think you are crying wolf. It largely comes down to being mindful of the bad things that can happen while being enthusiastic about all the good things that can come from a new idea or feature.
It can be tough not be the guy in the street with the end of the world sign when in a way it is part of your job, but learning how to find the balance is one of the challenges of being a sysadmin.