A while ago we decided that running our own DNS system was the best approach to take. We did this for many of the same reasons that our development team decided to move off of Window Azure.  One of our big gripes with our DNS hosting provider had been ease of management, simply put – a web interface is just not that good at managing a large number of DNS entries. They did have an API that allowed us to script some things, but it really wasn’t as robust as we needed it to be. It was a time consuming, repetitive process that didn’t need to be as bad as it was. So, today I present to you our three (semi)new DNS Servers!

  • ns1.serverfault.com
  • ns2.serverfault.com
  • ns3.serverfault.com

We have laid out our DNS servers in both of our data centers - Oregon and New York. We currently have two servers serving out of the New York data center, and have re-purposed one of the Web Tier servers in Oregon to act as our third DNS server. This layout – while not perfect (we could have a fourth server in Oregon to help us if NY goes down) – splits our servers geographically and logically.

NS1 and NS2.serverfault.com

  •  Dual Intel Xeon E5640 processors
  • 16 GB RAM

NS3.serverfault.com

  • Intel X3360 processor
  • 8 GB RAM

All three of our servers are running CentOS 5.5 and Bind 9.3.  When I did load testing before putting these servers into production I was able to get a max of about 2000 Request per second from the NY servers, and about 200 Requests per second out of the Oregon server. This is well within our growth zone, and when we do need to upgrade, it will be the Oregon server first giving us plenty of headroom to grow.

As you can see below we get just over 40 rps during peak times – well this is the holiday season, so it’s  a little low. Our normal peak is about 50-80 rps

The change went pretty well on our end, one minor hick-up with me putting a bad wild-card in place, and then a strange problem where a month after the transition we where still seeing 2-3rps on our Dynect reports. Even though we were leaving them Dynect was responsive, and they tracked the issue down to OpenDNS caching things for way too long. Once that was cleared up we were able to turn off Dynect.

We see this as one more step on the way to reaching or goal of making the servers that the Stack Exchange network runs on the best and most responsive infrastructure that is can be.

Any programmer worth their salt has used and praised revision control systems. The weird thing to me though is the fact that it seems as if the majority of sysadmins havn’t embraced revision control.  If you look at what you do as a Sysadmin, it really isn’t that much different from what programmers do. I’ll bet you spend more time scripting solutions so that you don’t have to do the same task over and over again.  I’ve worked both with and without VCS’s, and I can tell you that working without them is a horrible way to work. If you don’t have some sort of VCS to put all of your scripts into you quickly end up in the untenable situation where you have random scripts laying all over the place. You’ll have some scripts that only exist on the server they were written for, some scripts sitting out on the file share, while others are just sitting on one of your admin’s computers. Or, very possibly,  just sitting on your desktop!

I’ll be the first to admit, Sysadmins generally don’t need all the fancy features of some of the modern VCS’s. But, we do need something to centrally manage our scripts and config files – these really our life blood as Sysadmins.

When looking for a solution to implement, there are a plethora of  options out there to choose from:

  • CVS
  • Subversion
  • Git
  • Mercurial
  • TFS

All of these different systems have their ups and downs, a deep dive into the details of each is well beyond the scope of this post. I’ll just leave you with this bit of advice that has always been good to me – “Look at all the options and choose what is best for you.” Personally I’ve been very happy with just plain old CVS and that has served me very well. However, like any of the systems you need to get used to its little quirks. Once you choose one and get it running the way you want, the world gets much much brighter.

Take the following scenarios as an example and learn from them – I know all of you out there have run into a similar situation in the past. These very types of situations are drawn from my own experiences as a sysadmin, and have molded , and even changed my personal view on Version Control systems.

You write a little utility script for a seriously one-off function that your boss swears up and down you’ll never have to do again.

Point 1. Yes you will need to do this again because your result will be expected at every board meeting from now to infinity.

Point 2. You are going to think of a way to make it even more awesome and will completely refactor the script to make it more awesome.

Now, lets tackle the first situation erm first.

You’ve just tossed this script together on your local machine (sans comments, or even a unified coding style of course)  you run it and get the results your boss has been clamoring for all day (he even got out the pointy stick ). Your boss is happy because he has the info he needed to make his boss happy, you are happy because he put the pointy stick away before causing permanent damage. You move on to working on more important things like figuring out why the HA cluster keeps failing over at noon on Fridays. Three months later your boss comes around 1 hour before the next board meeting asking for “that thing you did last time with the thing.” I mean you should be able to pull it right up since you did the hard work for him 3 months ago right?  Ahh you say, I still have that script! this should only take 5 mins. Of course what you forgot is that you re-imaged that laptop you had the only copy of the script on, of course you backed your laptop up like a good sysadmin before re-imaging – it’s on one of those …. usb .. drives… over … there.

Now, you poor soul if you had checked your script into your “random stuff” VCS just in case you could have just pulled the most recent version down, run the script and forgotten about it for three more months.

Next, we have the much more probable situation where you write the same script as above, and decide “I can make this more awesome!” So you start hacking away. tweaking this and that moving this, removing that. Then suddenly you realize you have completely forgotten what you just changed that makes the script drop that image of your dog into the output of everything instead of the data you need. You start pushing that code, tweaking that bit there, move this around delete that. And now … and now the script does ….. nothing. You have no way back. You need to start over.

Yet another situation where a VCS system would have saved your skin. You could just simply have reverted to the last working version and moved on from there.

Beyond those – obviously made up – situations, a few more of the added benefits of a VCS system are:

  • Centralize backups of your scripts, configs, anything you put into the VCS
  • Check and see who checked in that change that broke the production environment
  • Persist your scripts easily across machines and employees
  • Feel smug in the fact you do this when you hear of someone losing “the script”

What this is really all about is preserving your hard work. You work hard to preserve the work of others, why not work hard to preserve your own work?

Just a quick note, we have moved blog.serverfault off of the Tumblr platform and onto a WordPress based self-hosted solution. And, no this has nothing to do with the outage that recently happend. We had been wanting to do this for a while, the outage just made us re-prioritize things.

I know we have been a little lacking in the posts recently, but this will change soon. We have been very busy working hard to make the Stack Exchange network better, stronger, and faster. I promise we will have more insightful posts coming soon!

The standard way to measure uptime is to measure the percent of time that your application is available over a period of time. This is always a good idea to do, but I don’t think it is the best way to measure the availability of your application for many companies. I think a better primary way is to think about how confident you are, your coworkers, and customers are in the availably of your application. If everyone is confident that your products will be up pretty much all the time then I think that this is a better measure than any “nines”.

I admit that going on how you feel as a primary measurement may seem a bit silly or touchy-feely. But I think your feelings and intuition solves a lot of issues with measuring uptime naturally. When even trying to figure out what a good uptime target is in percent of uptime some things you have to consider are:

Do you include maintenance time? This is probably more of an SLA issue than it is a practical one. For some sites, being down during off hours really isn’t a big deal. As an example, imagine an application that serves grade schoolers in the United States — it really doesn’t matter if it is down in the middle of the night.

Financial cost of achieving uptime vs cost of downtime? Some companies can map this pretty easily — “we loose X dollars per 30 seconds of downtime”. You then figure out the man power and equipment cost of high availability and all you need is some basic math skills. However, lots of companies are not finance or commerce companies so this is more difficult to measure. Take stackoverflow.com for example. We do have advertisers and our careers product, but our most valuable factor in my opinion is our users and our communities. How do you measure this other than keeping close ties to your users and see how they feel?

Development speed vs uptime? It seems intuitive that a fast development cycle will lead to more interruptions than a more careful and tested process. This might not be true, faster release cycles often mean smaller changes and might in the end cause less problems. I am not qualified to answer this, but it is another factor in uptime. If we accept this as true (I don’t, personally), then you have to weigh the cost of being down vs new features. The way this cost is measured is going to be difficult.

What counts as being down? If the site is slow, is this the same as being down? For example, at a certain point you might lose a customer to another commerce site if it is taking forever to load the purchase page. This is related to how Google now includes latency of your site in their page rank system. If you have new serious bugs but the site is still up, is this basically the same as being down?

Recovery Time: I think one of the most important things in uptime is recovery time. The primary thought most people have is “How do I keep from going down?”. Things go down no matter what, Facebook and Google have had some uptime issues before and everyone will. However, how much downtime you have when you do go down is a matter of how fast you can recover. I think recovery speed is a better place to start than attempting to eliminate going down at all. Of course, you need to do both.

The CHI of uptime: So if you are going to capture availability in a number it probably needs to include the above as factors in your equation as well as other things I probably haven’t even thought of. I heard a talk by Dharmesh Shah at Business of Software 2010 about measuring his customer’s happiness by factoring in various measurable stats as a Customer Happiness Index (CHI). (You can read about this talk here). Trying to do this with uptime would need to capture the above issues and is a worthy goal. However, getting there is not a small goal either.

Getting a good feeling for how confident your customers and employees are about the availability of your application captures all of these issues naturally. Uptime stats have their value, but I think numbers and stats are often overrated.