DNS at Stack Overflow is… complex.  We have hundreds of DNS domains and thousands of DNS records. We have gone from running our own BIND server to hosting DNS with multiple cloud providers, and we change things fairly often. Keeping everything up to date and synced at multiple DNS providers is difficult. We built DnsControl to allow us to perform updates easily and automatically across all providers we use.

The old way

Originally, our DNS was hosted by our own BIND servers, using artisanal, hand crafted zone files. Large changes involved liberal sed usage, and every change was pretty error prone. We decided to start using cloud DNS providers for performance reasons, but those each have their own web panels, which are universally painful to use. Web interfaces rarely have any import/export functionality, and generally lack change control, history tracking, or comments. We quickly decided that web panels were not how we wanted to manage our zones. 

Introducing DnsControl

DNSControl is the system we built to manage our DNS. It permits “describe once, use anywhere” DNS management. It consists of a few key components:

  1. A Domain Specific Language (DSL) for describing domains in a single, provider-independent way.
  2. An “interpreter” application that executes the DSL and creates a standardized representation of your desired DNS state.
  3. Back-end “providers” that sync the desired state to a DNS provider.

At the time of this writing we have 9 different providers implemented, with 3 more on the way shortly. We use it to manage our domains with our own BIND servers, as well as Route 53, Google Cloud DNS, name.com, Cloudflare, and more.

A sample might look like this description of stackoverflow.com:

D(“stackoverflow.com”, REG_NAMEDOTCOM, DnsProvider(R53), DnsProvider(GCLOUD),
    A(“@”, “198.252.206.16”),
    A(“blog”, “198.252.206.20”),
    CNAME(“chat”, “chat.stackexchange.com.”),
    CNAME(“www”, “@”, TTL(3600)),
    A(“meta”, “198.252.206.16”)
)
This is just a small, simple example. The DSL is a fully-featured way to express your DNS config. It is actually just javascript with some helpful functions. We have an examples page with more examples of the power of the language.

Running “dnscontrol preview” with this input will show what updates would be needed to bring DNS providers up to date with the new, desired, configuration. “dnscontrol push” will actually make the changes.

This allows us to manage our DNS configuration as code. Storing it this way has a bunch of advantages:

  • We can use variables to store common IP addresses or repeated data. We can make complicated changes, like failing-over services between data centers, by changing a single variable. We can activate or deactivate our CDN, which involves thousands of record changes, by commenting or uncommenting a single line of code.
  • We are not locked into any single provider, since the automation can sync to any of them. Keeping records synchronized between different cloud providers requires no manual steps.
  • We store our DNS config in git. Our build server runs all changes. We have central logging, access control, and history for our DNS changes. We’re trying to apply DevOps best practices to an area that has not seen those benefits so much yet.

I think the biggest benefit to this tool though is the freedom it has given us with our DNS.  It has allowed us to:

  • Switch providers with no fear of breaking things. We have changed CDNs or DNS providers at least 4 times in the last two years, and it has never been scary at all.
  • Dual-host our DNS with multiple providers simultaneously. The tool keeps them in sync for us.
  • Test fail-over procedures before an emergency happens. We are confident we can point DNS at our secondary datacenter easily, and we can quickly switch providers if one is being DDOSed.

DNS configuration is often difficult and error-prone.  We hope DnsControl makes it easy and more reliable. It has for us.

Some resources:

«
  • Oli

    This sounds somewhat like Terraform (https://www.terraform.io). Did you guys considering implementing this as patches/providers for Terraform?

    • Jerm

      The same question came up at SRECon.. they wrote this before TF existed.

      edit to add: they’re just now open-sourcing it

      • Oli

        I guess it shows the benefit of open sourcing early and often then – the Terraform guys could have used it!

    • Craig Peterson

      We hadn’t heard of TF is the answer. But really, Terraform looks to be much more general-purpose, while DNSControl is very focused on doing DNS well in a provider-independent way.

      I have not used TF yet, so I’m not sure how it works, but their example shows top level functions like “resource “dnsimple_record” “hello””, which is not how we represent things. Rather we say, “We have an A record for ‘hello’, and by the way, please host it with dnsimple”. I think that distinction is important.

      • Oli

        Yeah, I sort of realised that after I posted. Essentially it would be a provider independent resource in TF. I might see if they’re planning on anything like that.

  • Charlie Hayes

    Why write a DSL? This looks like C-style code and could probably function as such.

    • Craig Peterson

      We didn’t! Its really just javascript with some helper functions predefined. We call it a DSL because it generally follows a certain pattern, and looks a certain way. Our config is more complicated than a simple config language like json can accomidate, so once we started hacking in variables and things we decided to just use a real programming language for it.

      • Charlie Hayes

        I think using JS is a fine thing to do, but if it’s JS, why call it a DSL?

  • Lena kogan

    Very interesting, how do you handle Apex records?

  • chen

    so the dnsconfig.js is basically a long file describing all the domains and their records that are shared between your DNS providers? Isn’t that a very long file to maintain?

    I separate my domains by internal and external. to manage the internal domains i use Bind and for the external I use DYN and NS1. some of the domains are shared between the 3 providers. If i understand correctly, the domains that I manage under all 3 providers show up in the dnsconfig file once with 1 registrar and 3 providers, and the rest of the domains will have 1 registrar and only 1 provider?

    • Neil L

      I’d love to know the answer to this question. If I have to put my hundreds of domains and thousands of records in one file (dnsconfig.js), that’s a dealbreaker.

    • Chen

      also, does it support geo its or just ordinary DNS records?

  • chen

    Another question: What about dynamic changes made to dns by external systems such as provisioning systems and CM systems? they directly make changes to bind zone files. how would that effect the dnsconfig file (if at all) and wouldn’t it cause the dnsconfig file to be different from the actual zone file and therefore break the dnsconfig file/cause the dnscontrol to delete new records that aren’t in the dnsconfig file?