DNS at Stack Overflow is… complex.  We have hundreds of DNS domains and thousands of DNS records. We have gone from running our own BIND server to hosting DNS with multiple cloud providers, and we change things fairly often. Keeping everything up to date and synced at multiple DNS providers is difficult. We built DnsControl to allow us to perform updates easily and automatically across all providers we use.

The old way

Originally, our DNS was hosted by our own BIND servers, using artisanal, hand crafted zone files. Large changes involved liberal sed usage, and every change was pretty error prone. We decided to start using cloud DNS providers for performance reasons, but those each have their own web panels, which are universally painful to use. Web interfaces rarely have any import/export functionality, and generally lack change control, history tracking, or comments. We quickly decided that web panels were not how we wanted to manage our zones. 

Introducing DnsControl

DNSControl is the system we built to manage our DNS. It permits “describe once, use anywhere” DNS management. It consists of a few key components:

1. A Domain Specific Language (DSL) for describing domains in a single, provider-independent way.
2. An “interpreter” application that executes the DSL and creates a standardized representation of your desired DNS state.
3. Back-end “providers” that sync the desired state to a DNS provider.

At the time of this writing we have 9 different providers implemented, with 3 more on the way shortly. We use it to manage our domains with our own BIND servers, as well as Route 53, Google Cloud DNS, name.com, Cloudflare, and more.

A sample might look like this description of stackoverflow.com:

D(“stackoverflow.com”, REG_NAMEDOTCOM, DnsProvider(R53), DnsProvider(GCLOUD),
    A(“@”, “198.252.206.16”),
    A(“blog”, “198.252.206.20”),
    CNAME(“chat”, “chat.stackexchange.com.”),
    CNAME(“www”, “@”, TTL(3600)),
    A(“meta”, “198.252.206.16”)
)

This is just a small, simple example. The DSL is a fully-featured way to express your DNS config. It is actually just javascript with some helpful functions. We have an examples page with more examples of the power of the language.

Running “dnscontrol preview” with this input will show what updates would be needed to bring DNS providers up to date with the new, desired, configuration. “dnscontrol push” will actually make the changes.

This allows us to manage our DNS configuration as code. Storing it this way has a bunch of advantages:

• We can use variables to store common IP addresses or repeated data. We can make complicated changes, like failing-over services between data centers, by changing a single variable. We can activate or deactivate our CDN, which involves thousands of record changes, by commenting or uncommenting a single line of code.
• We are not locked into any single provider, since the automation can sync to any of them. Keeping records synchronized between different cloud providers requires no manual steps.
• We store our DNS config in git. Our build server runs all changes. We have central logging, access control, and history for our DNS changes. We’re trying to apply DevOps best practices to an area that has not seen those benefits so much yet.

I think the biggest benefit to this tool though is the freedom it has given us with our DNS.  It has allowed us to:

• Switch providers with no fear of breaking things. We have changed CDNs or DNS providers at least 4 times in the last two years, and it has never been scary at all.
• Dual-host our DNS with multiple providers simultaneously. The tool keeps them in sync for us.
• Test fail-over procedures before an emergency happens. We are confident we can point DNS at our secondary datacenter easily, and we can quickly switch providers if one is being DDOSed.

DNS configuration is often difficult and error-prone.  We hope DnsControl makes it easy and more reliable. It has for us.

Some resources:

• The github repo
• The documentation site (with Getting Started Guide)
• A talk I gave at SRECon 2017 Americas

A while ago we decided that running our own DNS system was the best approach to take. We did this for many of the same reasons that our development team decided to move off of Window Azure.  One of our big gripes with our DNS hosting provider had been ease of management, simply put – a web interface is just not that good at managing a large number of DNS entries. They did have an API that allowed us to script some things, but it really wasn’t as robust as we needed it to be. It was a time consuming, repetitive process that didn’t need to be as bad as it was. So, today I present to you our three (semi)new DNS Servers!

• ns1.serverfault.com
• ns2.serverfault.com
• ns3.serverfault.com

We have laid out our DNS servers in both of our data centers – Oregon and New York. We currently have two servers serving out of the New York data center, and have re-purposed one of the Web Tier servers in Oregon to act as our third DNS server. This layout – while not perfect (we could have a fourth server in Oregon to help us if NY goes down) – splits our servers geographically and logically.

NS1 and NS2.serverfault.com

•  Dual Intel Xeon E5640 processors
• 16 GB RAM

NS3.serverfault.com

• Intel X3360 processor
• 8 GB RAM

All three of our servers are running CentOS 5.5 and Bind 9.3.  When I did load testing before putting these servers into production I was able to get a max of about 2000 Request per second from the NY servers, and about 200 Requests per second out of the Oregon server. This is well within our growth zone, and when we do need to upgrade, it will be the Oregon server first giving us plenty of headroom to grow.

As you can see below we get just over 40 rps during peak times – well this is the holiday season, so it’s  a little low. Our normal peak is about 50-80 rps

The change went pretty well on our end, one minor hick-up with me putting a bad wild-card in place, and then a strange problem where a month after the transition we where still seeing 2-3rps on our Dynect reports. Even though we were leaving them Dynect was responsive, and they tracked the issue down to OpenDNS caching things for way too long. Once that was cleared up we were able to turn off Dynect.

We see this as one more step on the way to reaching or goal of making the servers that the Stack Exchange network runs on the best and most responsive infrastructure that is can be.