A while ago we decided that running our own DNS system was the best approach to take. We did this for many of the same reasons that our development team decided to move off of Window Azure.  One of our big gripes with our DNS hosting provider had been ease of management, simply put – a web interface is just not that good at managing a large number of DNS entries. They did have an API that allowed us to script some things, but it really wasn’t as robust as we needed it to be. It was a time consuming, repetitive process that didn’t need to be as bad as it was. So, today I present to you our three (semi)new DNS Servers!

  • ns1.serverfault.com
  • ns2.serverfault.com
  • ns3.serverfault.com

We have laid out our DNS servers in both of our data centers – Oregon and New York. We currently have two servers serving out of the New York data center, and have re-purposed one of the Web Tier servers in Oregon to act as our third DNS server. This layout – while not perfect (we could have a fourth server in Oregon to help us if NY goes down) – splits our servers geographically and logically.

NS1 and NS2.serverfault.com

  •  Dual Intel Xeon E5640 processors
  • 16 GB RAM

NS3.serverfault.com

  • Intel X3360 processor
  • 8 GB RAM

All three of our servers are running CentOS 5.5 and Bind 9.3.  When I did load testing before putting these servers into production I was able to get a max of about 2000 Request per second from the NY servers, and about 200 Requests per second out of the Oregon server. This is well within our growth zone, and when we do need to upgrade, it will be the Oregon server first giving us plenty of headroom to grow.

As you can see below we get just over 40 rps during peak times – well this is the holiday season, so it’s  a little low. Our normal peak is about 50-80 rps

The change went pretty well on our end, one minor hick-up with me putting a bad wild-card in place, and then a strange problem where a month after the transition we where still seeing 2-3rps on our Dynect reports. Even though we were leaving them Dynect was responsive, and they tracked the issue down to OpenDNS caching things for way too long. Once that was cleared up we were able to turn off Dynect.

We see this as one more step on the way to reaching or goal of making the servers that the Stack Exchange network runs on the best and most responsive infrastructure that is can be.

«
»
  • I know Amazon Route 53 came a little late in your planning and deployment cycle, but it seems to have a powerful API, much like most of their cloud, and is very well distributed across the world. Did you evaluate them at all?

    • I did look at it briefly when it came out. We had already deployed our internal servers so I didn’t look at it too seriously. However, some quick math and the AWS calculator shows that it really doesn’t make sense for a large site. Just as an example, it would cost us (~150req/s) ~204$/mo which is ~2448$/yr and ~12240$ over the five year typical lifecycle of a server. And that is with our current request rate, it will only get worse. Just as an example our servers cost about 3-4k each, and figure 1k in bandwidth charges over 5 years thats about 3k less over a five year lifecycle. That gap just gets bigger as our trafic grows. lets assume a ~300req/s rate – at which we need to buy lets call it two more servers at 4k ea – that is 8k + 8k original investment. and call it 2k in bandwidth and power. that would be ~400$/mo ~4800$/yr and ~24000 over a 5 year lifecycle. So to double our traffic it would be almost 3x more expensive over the lifecycle of a server to go with AWS. And just for fun some really big numbers lets say we have 4 of the new servers (2 more in addition to what we have ) that can handle ~2000 req/s. at ~1500 req/s (being safe) * 4 servers that is ~ 6000 req/s that would cost us: ~4100$/mo; $49000$/yr; and ~246000$/5 year lifecycle vs. our ~16k one time cost and call it 10 k over five years for power and bandwidth. So I guess my point is AWS and other “cloud” based services are a great boon for small and medium sites that don’t have the capital or the recourses to run a great infrastructure, but they don’t scale up when you look at the raw numbers. Even the best API isn’t worth that much. deep breath ok I havn’t written that down before … I might have to extrapolate and turn the above into a real blog post later 🙂

      • I would really like to see that blog post at some point. But if you do a real cost analysis, Amazon has typically pushed down prices over the long run, if we take EC2, S3, and others as a model for what is going to happen with Route 53.

        Plus it is hard to say that it is best for small to medium sites, because some how large companies and governments have justified it. For example Netflix and the U.S. Government are moving or have already moved large parts of their infrastructure over to Amazon. Twitter runs their CDN from Amazon, and rumored parts of their backed infrastructure too.

        So at some point these large sites decided that it was cheaper to move to Amazon over building data centers, and hiring swaths of admins to manage them. That is why I would love to see a blog post on this, because I don’t think you have factored your own time in to the equation which is usually the largest portion of any server deployment. Also I don’t think you have factored the ability to scale, if you ever had to add another server, or maybe put one over in Sweden or something to handle the European traffic.

        Looking forward to the future post.

  • Wim

    Just curious, but why does a dual quad-core Core2 box get 10x the requests rate of a dual six-core Westmere box? (Assuming mem, OS, network are all the same?)

    • Sorry, copy-paste fail. The Westmere’s are in NY in the new servers. I’ve updated the post.

  • Ed Bynum

    I know next to nothing about Bind, but 16GB of memory seems like it would handle an awfully large DNS Zone. At 2000 rps, and assuming (maybe incorrectly) that requests close pretty immediately when they’re complete, that means you have 8MB/request (if memory is the limiting factor). That seems absurdly large. Am I confused? Is it just a standard configuration? Are you using the memory with a front-end cache (memcached?) for your webservers?

  • You mentioned OpenDNS – where are you using it? 🙂