Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.
At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.
There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.
Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.
Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.
Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.
We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.
At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.
Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?
Why Use Virtualization?
Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.
This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:
- Live migration (vMotion in VMWare terms)
- Running multiple operating systems (i.e. Windows and Linux) on the same hardware
- Easier to get full utilization of hardware resources by moving VMs around
These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?
The Joy of Commodity Hardware
As Wikipedia defines it:
“Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”
Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.
So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.
Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.
Virtualization and Centralized Storage are Best Friends
With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.
Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.
Virtualized Clusters to the Rescue?
If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.
VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.
Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.
What Will We Go With?
Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.
One website defines production control as the following:
Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.
That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.
Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.
Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.
As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.
An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.
For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.
Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”
The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.
Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?
We have gone and done it, we’ve expanded our Systems Administration team to THREE people. I’d like to introduce you to Peter Grace who we have just brought on board as our third sysadmin.
This is the second company I’ve had the honor of working with Pete at, and I can say that he is an exemplary System Administrator. In addition to being a great sysadmin, Pete is a fellow gun enthusiast, which increases our zombie attack survival rate by 35%.
Pete and I have done many great things together in the past, and I’m very excited about what he brings to us and the great things he, Kyle and myself are going to do as we move forward doing our part to make the internet better.
I have always had the notion that some companies “Get It” when marketing to programmers, sysadmins, and hackers. A couple weeks ago I was in Silicon Valley at Tech Field Day listening to a lot of presentations with some fellow geeks. In some of these presentations they “got it”, in others they didn’t.
The fundamental difference is that the ones that get it tell me about their technology, and the ones that don’t tell me about their products. True geeks have one common trait, and that is that they like to learn how things work. We get excited about learning and news ideas. Exciting technologies solve problems in non-obvious ways. Since these are not obvious, we want to learn about them, and in as much detail as possible.
Products, on the other hand, are how these ideas are packaged. For the most part, we don’t care. If we do care, that comes later, only if it will solve a problem that we have will we start to care about implementation and packaging. The advantage of telling us about your technology is that we do care, even if we don’t have a use for it at the moment, because we are geeks – it is our nature. When done right, we will then associate your product with the technology and that will be enough.
This has some consequences that marketing should be aware of when targeting tech people. First, we generally don’t want to talk to you directly, at least, not for very long. This is not because you are not important or interesting, it is just that you probably can’t get us excited about your technology like your engineers can. You can enable your engineers to present your technology well, and that is what people who are good at geek marketing do. If you can’t get your actual engineers to present no mater how hard you try, your screwed. It’s not your fault, your company just sucks and you probably just need to move on. Lastly, everything targeted to tech people should be aimed at getting us excited about your technology, not your product.
With my recent experiences at tech field day there were some good examples of this done right. Pure Storage taught me about why I see SSDs fail and a new type of RAID they invented suited for SSDs. Arkeia educated me about various implementations of deduplication in backups. Data Direct Networks introduced me to the concept of object store filesystems. However, in that case I wanted to learn even more given that amount of time. How successful they were came down to how much time they all devoted to fulfilling my desire to learn more about technology vs. telling me about their products.
I’ve recently been looking back on what we have written about our architecture in the past, and came to a stunning realization. That realization is that while we have many many different articles about what we have been doing there hasn’t been a good, solid overview of our architecture in a long time. In fact, the last really comprehensive write-up was done by Jeff before this blog even existed. And, boy I do have to say there has been quite a lot of change behind the scenes since then. So, my dear readers I’m going to take some time – and my next few blog posts – to give everyone an in depth look into how we have the Stack Exchange Network setup to serve between 12 and 14 Million page views per day.
How these posts will breakdown
Since we have obviously grown, and are offering more services to our users I’m going to break these posts out by each of the 4 major services we offer to our user base:
- Core Q&A (this includes the API)
- Community Blogs
Each one of these systems all work towards our goal of making the internet better, but they have different requirements and different challenges.
In this first post, I’ll be focusing on our core Q&A system, since that is after all our bread and butter.
First, a high level overview of how everything is put together:
Our core hardware setup hasn’t changed all that much. Well, I should say the chassis haven’t changed that much. We’ve done a lot of work to upgrade the internals of the servers when needed to address performance issues as they came up, as well as handle issues that resulted from Stack Overflow being so big.
Of these 10 Servers, 3 are dedicated to Stack Overflow with an additional 3 servers serving Stack Overflow and the Stack Exchange Network. We have one server dedicated to Dev/QA – which also hosts meta.stackoverflow.com. Our Web Tier machines normally operate between 5 and 20% utilization. We have plenty of room to grow on these boxes.
- 10 Dell R610 IIS web servers:
- 2x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
- 16 GB RAM
- Windows Server 2008 R2
- 2 drives
- RAID 1
- 2x Intel 320 300GB SSD (RAID 1)
We have two database server pairs. One pair is dedicated to running Stack Overflow, and the other runs the rest of the network. We run development against the secondary server of the non-stack overflow database pair. Both of our database pairs run at about 20% utilization, so once again we have room to grow here as well.
- 2 Dell R710 database servers:
- 2x Intel Xeon Processor X5680 @ 3.33 GHz
- 96 GB RAM
- 8 spindles
- Mirrored Pair for OS
- 6 disk RAID10 for databases
- SQL Server 2008 R2 SP1
- 2 Dell R710 database servers (Stack Overflow Dedicated):
- 2x Intel Xeon Processor X5680 @ 3.33 GHz
- 96 GB RAM
- 8 drives
- Mirrored Pair for OS
- 6 drive RAID10 of Intel X25-E SSDs for Database
- SQL Server 2008 R2 SP1
We run redundant Redis servers for our caching tier.
- 2 Dell R610 Redis servers:
- 2x Intel Xeon Processor E5640 @ 2.66 GHz
- 16 GB RAM
We use HAProxy for our load balancing, and Cisco Switching.
- 2 Dell R610 HAProxy servers:
- 1x Intel Xeon Processor E5640 @ 2.66 GHz
- 4 GB RAM
- Ubuntu Server
- 6 WS-C2960S-48TS-L Gigabit switches
- FlexStack (two stacks, 4 switches and 2 switches)
As with any system, making sure that your data is backed up and the backups are good is an integral part to your service offering. We backup our databases nightly and restore them to two different locations. One local to our NY data center for our devs to work against, and one remote in our OR data center.
Overall I believe that we are in a good place and have plenty of room to grow given our current setup. As always we will constantly be looking at our infrastructure and tweaking it to get the best performance possible and give our users the best experience possible.
System administrators can sometimes seem like they are trying to get everyone down. Someone goes to them with a great idea, application, feature, or just something they just want to get done. When the sysadmin comes in, instead of being enthusiastic about the great idea, he starts talking about viruses, hacking, earthquakes, floods, fires, Vogon invasions, and all the things that can go wrong. Because of this, the natural reaction is to stick the sysadmin in the basement and stop inviting them to parties.
In order to work past this issue, it takes a little work from both sides of the table. Coworkers need to remember that disaster is part of a sysadmin’s job and that Murphy’s law is part of what they do. Part of a sysadmin’s job to think about this stuff all the time because we are expected to have backups and to be able to recover from small events to major disasters. Major disasters are uncommon but do happen, hence the importance of offsite backups at a minimum. Small disasters, such as power failures, router failures are actually quite common. When a sysadmin is doing a really good job, people don’t even know about some of the smaller failures.
It also falls on the sysadmin’s shoulders not get stuck in this gloom and doom world. We have remind ourselves not to get in the habit of saying No to everything because something could go wrong. This is important because the danger is that that people will try to start bypassing you or just not believing you when you talk about very real threats because they think you are crying wolf. It largely comes down to being mindful of the bad things that can happen while being enthusiastic about all the good things that can come from a new idea or feature.
It can be tough not be the guy in the street with the end of the world sign when in a way it is part of your job, but learning how to find the balance is one of the challenges of being a sysadmin.
Our CEO Joel Spolsky wrote a book about hiring programmers that is titled Smart & Gets Things Done. The idea behind the title is that when it comes down to it, you really just want to hire someone who is both smart and will get things done:
“People who are smart, but don’t get things done … would rather mull over something academic about a problem than ship on time”.
This applies equally in the area of system administration. Our CTO Jeff Atwood recently tweeted a reference to this:
When I read that I felt like he was talking to the sysadmin team (I hope he wasn’t — but the fact that I felt like he might have been is enough). The reason for this isn’t that we are not getting things done; I think we work hard and get a lot of things done. Rather, I think we have tendency to pick the interesting tasks over the uninteresting or tedious tasks.
Assuming as a system administrator you actually like what you do, I think tasks end up looking like the following:
So there are some interesting tasks that would be good to do, but may not need to be done. Then you have most tasks, which if you like your job, both need to be done and are interesting. Lastly, you have the tasks that are not interesting but do need to be done.
With a tendency to give too much weight to the interesting tasks or pieces of a project, that category on the right will grow over time. I like to think of keeping that category under control as eating my vegetables. Ideally, one maintains a good balanced diet all the time. But in reality there are times when you need to get on the scale and come to grips with the fact that you might need to go on a healthy diet for a while.
Lastly, don’t worry, the irony of taking some time to write this post is not lost on me.
This Saturday, August 6th starting at 3pm EDT (19:00 UTC) we will be taking our sites offline for maintenance. We are aiming to keep the downtime to about one to four hours. Should we need to extend this window I will post updates on this post.
During this window we are going to be patching our database servers. We are also migrating our servers to a new AD domain so we have everything under a consistent naming structure.
Hello, our name is Stack Exchange and we have an alerting problem. It hurts us, our friends, and our family. We are not sure how we got here. Sure, we get some extraneous alerts, but everyone does right? Then one day we woke up and had an inbox full of alerts. We wrote it off. We told ourselves that it happens to everyone. But then it happened again, and then again. We don’t want this. We don’t want to live like this anymore. We are ready to pick ourselves up. We are ready to face this problem and live a new, and better life.
Don’t know if you have a problem? Here are some of the signs:
- You get alerts that you just don’t care about, because of this you maybe don’t see the ones you do care about.
- The more serious ones wake us up in the middle of the night when they don’t need to because someone else is already dealing with them.
- When something major happens your inbox is flooded.
- You set up email rules to handle them.
- You are ashamed.
If like us, you have an alerting problem and you have admitted it, I believe finding the righteous path starts with one rule:
Every Alert Requires Action
Every alert requires action. The problem we have right now at Stack Exchange is that alerts don’t require that we do anything. If we are to address our alerting problem — I believe this, more than anything, needs to be fixed. When I mean every alert, I do mean every alert. So what sort of actions can we take:
- If it is real problem and you are dealing with it — acknowledge the alert.
- If it is a false alert, acknowledge and adjust the threshold level
- If the alert was a flood of alerts, acknowledge them and set up dependencies.
In order to do this we need a few things. We need a system that allows to effectively acknowledge alerts without too much friction. We need cooperation from everyone to use this system once it is in place. Lastly, we need to accept that we don’t have the power to cure our alerting problem. We can, however, through constant vigilance, get it under control.