A while ago we decided that running our own DNS system was the best approach to take. We did this for many of the same reasons that our development team decided to move off of Window Azure. One of our big gripes with our DNS hosting provider had been ease of management, simply put – a web interface is just not that good at managing a large number of DNS entries. They did have an API that allowed us to script some things, but it really wasn’t as robust as we needed it to be. It was a time consuming, repetitive process that didn’t need to be as bad as it was. So, today I present to you our three (semi)new DNS Servers!
We have laid out our DNS servers in both of our data centers - Oregon and New York. We currently have two servers serving out of the New York data center, and have re-purposed one of the Web Tier servers in Oregon to act as our third DNS server. This layout – while not perfect (we could have a fourth server in Oregon to help us if NY goes down) – splits our servers geographically and logically.
NS1 and NS2.serverfault.com
- Dual Intel Xeon E5640 processors
- 16 GB RAM
- Intel X3360 processor
- 8 GB RAM
All three of our servers are running CentOS 5.5 and Bind 9.3. When I did load testing before putting these servers into production I was able to get a max of about 2000 Request per second from the NY servers, and about 200 Requests per second out of the Oregon server. This is well within our growth zone, and when we do need to upgrade, it will be the Oregon server first giving us plenty of headroom to grow.
As you can see below we get just over 40 rps during peak times – well this is the holiday season, so it’s a little low. Our normal peak is about 50-80 rps
The change went pretty well on our end, one minor hick-up with me putting a bad wild-card in place, and then a strange problem where a month after the transition we where still seeing 2-3rps on our Dynect reports. Even though we were leaving them Dynect was responsive, and they tracked the issue down to OpenDNS caching things for way too long. Once that was cleared up we were able to turn off Dynect.
We see this as one more step on the way to reaching or goal of making the servers that the Stack Exchange network runs on the best and most responsive infrastructure that is can be.