DNS at Stack Overflow is… complex.  We have hundreds of DNS domains and thousands of DNS records. We have gone from running our own BIND server to hosting DNS with multiple cloud providers, and we change things fairly often. Keeping everything up to date and synced at multiple DNS providers is difficult. We built DnsControl to allow us to perform updates easily and automatically across all providers we use.

The old way

Originally, our DNS was hosted by our own BIND servers, using artisanal, hand crafted zone files. Large changes involved liberal sed usage, and every change was pretty error prone. We decided to start using cloud DNS providers for performance reasons, but those each have their own web panels, which are universally painful to use. Web interfaces rarely have any import/export functionality, and generally lack change control, history tracking, or comments. We quickly decided that web panels were not how we wanted to manage our zones. 

Introducing DnsControl

DNSControl is the system we built to manage our DNS. It permits “describe once, use anywhere” DNS management. It consists of a few key components:

  1. A Domain Specific Language (DSL) for describing domains in a single, provider-independent way.
  2. An “interpreter” application that executes the DSL and creates a standardized representation of your desired DNS state.
  3. Back-end “providers” that sync the desired state to a DNS provider.

At the time of this writing we have 9 different providers implemented, with 3 more on the way shortly. We use it to manage our domains with our own BIND servers, as well as Route 53, Google Cloud DNS, name.com, Cloudflare, and more.

A sample might look like this description of stackoverflow.com:

D(“stackoverflow.com”, REG_NAMEDOTCOM, DnsProvider(R53), DnsProvider(GCLOUD),
    A(“@”, “”),
    A(“blog”, “”),
    CNAME(“chat”, “chat.stackexchange.com.”),
    CNAME(“www”, “@”, TTL(3600)),
    A(“meta”, “”)
This is just a small, simple example. The DSL is a fully-featured way to express your DNS config. It is actually just javascript with some helpful functions. We have an examples page with more examples of the power of the language.

Running “dnscontrol preview” with this input will show what updates would be needed to bring DNS providers up to date with the new, desired, configuration. “dnscontrol push” will actually make the changes.

This allows us to manage our DNS configuration as code. Storing it this way has a bunch of advantages:

  • We can use variables to store common IP addresses or repeated data. We can make complicated changes, like failing-over services between data centers, by changing a single variable. We can activate or deactivate our CDN, which involves thousands of record changes, by commenting or uncommenting a single line of code.
  • We are not locked into any single provider, since the automation can sync to any of them. Keeping records synchronized between different cloud providers requires no manual steps.
  • We store our DNS config in git. Our build server runs all changes. We have central logging, access control, and history for our DNS changes. We’re trying to apply DevOps best practices to an area that has not seen those benefits so much yet.

I think the biggest benefit to this tool though is the freedom it has given us with our DNS.  It has allowed us to:

  • Switch providers with no fear of breaking things. We have changed CDNs or DNS providers at least 4 times in the last two years, and it has never been scary at all.
  • Dual-host our DNS with multiple providers simultaneously. The tool keeps them in sync for us.
  • Test fail-over procedures before an emergency happens. We are confident we can point DNS at our secondary datacenter easily, and we can quickly switch providers if one is being DDOSed.

DNS configuration is often difficult and error-prone.  We hope DnsControl makes it easy and more reliable. It has for us.

Some resources:

I’ve been getting some great response to my previous post, and I wanted to make a few things very clear.


What I’m not doing –

  1. I’m not advocating or dismissing any particular configuration management tool.
  2. I’m not discounting the tough work done by companies and community projects that have created abstractions on managing disparate systems.

What I am attempting to do –

  1. Highlight the challenges of cross-platform management and application management.
  2. Show one of the efforts in providing a standards based management abstraction.
  3. Offer my thoughts on why I see value in that direction and what challenges I see.

Managing the Operating System vs. Managing Applications

There is definitely some confusion around using CIM to manage the OS vs managing applications.

  1. CIM Classes can be used to model applications as well as OS resources.
  2. Most of the “applications” that are packaged as roles and features in Windows Server expose a CIM management API.
  3. WMI is an implementation of the CIM standard and starting with Server 2012 and the Windows Management Framework V3, CIM is exposed via WSMAN.

Hyper-V, File Shares, Clustering, IIS, and others all offer CIM based management models.  Other applications can expose a CIM management model as well.  As long as the host CIM server (WMI on Windows and OMI or OpenPegasus or ???? on Linux based Operating Systems) is operational, applications can also offer their configuration and status via that channel by creating a provider.  To do this on Windows, there is some documentation to get you started:

To do this with OMI, you can find some documentation and source at https://collaboration.opengroup.org/omi/documents.php.

The Current State – Reprise

But, but, tool {fill in your favorite tooling here} already does THAT!!!

There are a number of tools that valiantly strive to provide cross-platform management.  I mentioned several of them in my last post, but there are a number of others.

Yep, it does.  Until…

things change.  The challenge these tools have is that they have had to implement their abstractions against very different implementations.  The problem there is that these things are not stagnant.  The management APIs can change over time and since there is not a standard description of the API or underlying configuration.

If CIM were the standard API exposing the configuration, the underlying implementation details can change, but configuration management and monitoring tooling don’t have to care about that.  The tool vendors and community projects can focus on other value adds for their particular tooling, rather than being forced to continually update the basics.

The Next Steps

We are still in the early stages of the push for CIM and WSMAN.  We’ll have to see how adoption picks up.  The continuing work around OMI holds promise, but it needs a deployment or integration story for various Linux distros and more public providers for managing components of the Linux OS and attendant applications.

There have been some interesting announcements at TechEd in relation to Windows Server 2012 R2 (watch this video and pay attention around 49 minutes in).  I’m going to talk more about this feature and it’s implications and my plans with it in the very near future.


​Configuration management today is mess if you work in a heterogeneous platform.

There is tooling that takes a stab at it, and is getting better (from the *nix world – Puppet, CFEngine, and Chef and from the Windows world – System Center Configuration Manager, Group Policy, among other third party application deployment platforms).  These tools are all well and good, but they fall down when reaching across the OS divide.  Puppet, Chef, and CFEngine (there are others as well, but these are some of the more popular) all have some cross platform support, but it feels unnatural (especially in module or recipe development).

Why is this a mess?

Windows is traditionally described as having an API oriented management model, whereas *nix has a document based management model.

Well, that’s a load of crappy, crap, crap.  What does that actually mean?

It means that the two operating systems offer two different management models.  The two different models have different abstractions and idioms for operating system constructs.  Let’s look at a concrete example, setting a static IP on a network interface (just the rough strokes.. I’m not going to spend too much time on the minutia). As I stated before, Linux uses a document oriented management model, so to configure my network interface, I’ll edit a document or two.

The Linux (Centos) example:

  1. Find the correct interface file under /etc/sysconfig/network-scripts
  2. Open it in your text editor of choice
  3. Edit it to contain your desired settings for the network interface and save the file
  4. If you need to add/modify DNS servers, find /etc/resolv.conf
  5. When done, you can bring your interface online with a command line call to
    ifup eth0

You’ll have something that looks like this for your network configuration file:

And something like this for your resolv.conf:
domain serverfault.com
search serverfault.com
That wasn’t so bad, and as an added benefit, they are just text files, so I could check them in to a revision control system (Versioning FTW!). Now, let’s look at what we’d need to do on the Windows side.  Since this is a blog for a community of “professional” systems administrators, we are going to dispense with any GUI example for doing this.

The Windows Server (2008 R2) example:

  1. Use WMI to retrieve the network adapter interface index.
  2. Use WMI to retrieve the network adapter configuration by the index.
  3. Set the desired IP address, gateway, and DNS servers and suffix against the WMI object.

You can use the following PowerShell commands to make those changes:

$NetworkAdapter = Get-WMIObject Win32_NetworkAdapter -filter "NetConnectionID = 'Local Area Connection'"
$NetworkAdapterConfiguration = Get-WMIObject Win32_NetworkAdapterConfiguration -filter "InterfaceIndex = $($NetworkAdapter.InterfaceIndex)"

The Windows Server (2012) example:

  1. Set the desired IP address and gateway based on the interface name.
  2. Set the DNS servers and with a few more PowerShell commands.

You can use the following PowerShell commands to make those changes:

$IPAddressParameters = @{
            IPAddress = ''
            InterfaceAlias = 'Local Area Connection'
            AddressFamily = 'IPv4'
            PrefixLength = 24
            DefaultGateway = ''
Set-NetIPAddress @IPAddressParameters
Set-DNSClientServerAddress -InterfaceAlias 'Local Area Connection' -ServerAddresses '',''
Both of these examples are interactive commands, but I could easily save them in a file and place that under version control (and I should).

So what?

The examples don’t look all too different, but they do illustrate the difference between similar operations.  In both  examples, I end up with an artifact, but one is for a one time application of the setting (the Windows side) and the other is the setting storage location (the Centos example).

On the Centos box, we had to edit a file where the configuration was read from.  On the Windows servers, we updated settings via a WMI API (in both cases.. on Server 2012 there are more built in cmdlets, but many of them are thin wrappers over the WMI APIs) and not the actual end storage location.

This is what

Any configuration management tool that works in a cross platform capacity needs to understand these distinctions and check based on OS type which implementation to use when configuring a system.  This means for most configuration types, you’d have a big “IF” block where *nix based OS’s follow this line of processing and Windows based machines follow  the other line of processing.  This can become a maintenance nightmare as OS versions change the API on the Windows side or modify location and or layout of the configuration files on the *nix side.

And it’s even worse…

Now, what happens when you have a model that doesn’t translate across both worlds?

For example, how do manage file permissions?

Posix style permissions (used on most *nix variants) assigns permissions are nowhere as discreet as NTFS file permissions.  In addition on Windows, the file system auditing is also configured via the permissions configuration. In the reverse, on *nix files can be set as executable, where that is handled by file type mappings based on file extensions in Windows. This fragmentation leads to more complex implementations on the side of configuration management software developers or missing feature coverage.  In either case, this is a loss for the sysadmin who maintains cross platform environments.

But what if….

there was a common method of interacting with operating systems, regardless of what was running underneath? What if this method used a common transport (open standard) and communications were defined by an open standard? This is the direction Microsoft is taking with CIM and WS-Management.

CIM (Common Information Model) is a DMTF (Distributed Management Task Force) standard for describing management information for systems, networks, applications, and services.

WS-Management is another DMTF standard for management communication, focused on CIM traffic.

Microsoft has contributed to an open source project hosted by the Open Group called OMI.  OMI is a CIM server that communicates over WS-Management and is implemented to run on *nix based operating systems.

I’m personally interested in where this will go, given Microsoft’s market power (Cisco and Arista are working on incorporating OMI into their network switches).  The idea of a shared management model is very appealing to me, as I work in a cross platform environment.  I’m responsible for our Windows infrastructure, but I have to be able to work with our *nix infrastructure as well.  If I could use one model for interacting with both, that’s a huge win for me and my team.

This wouldn’t eliminate any domain specific knowledge on either OS side, as you’d still need to know what buttons to push and knobs to tweak to get things going and do some deep troubleshooting.  It does, however, make the idea standardizing how various OS components can be accessed, making basic configuration, monitoring, and troubleshooting much easier.

I’m interested because…

this pushes the implementation down to the OS provider (or the CIM provider provider) and gives vendors one target to hit for configuration standards.  In the Microsoft case, they can say “Follow this standard and any Windows system can manage you with minimal effort.”  If other OS’s support CIM and WS-MAN as well, it becomes easy to offer management interfaces there as well.

Obviously this would be huge change to the existing way of doing things for OS and application developers, not to mention systems administrators that are invested in their existing ways of doing things.

I don’t see another good alternative though, as the numbers and variety of systems continue to scale up and “cloud” becomes more of a factor in our environments, yet the number of admins is staying stagnant or being reduced.  Simplifying the management and monitoring surface makes sense in today’s and likely tomorrow’s data center landscape.

It doesn’t solve every problem and vendors can still implement vendor specific extensions (and we know how well that’s worked with SNMP).


NOTE – Be sure to check out some clarifications and expanded discussion in my followup post.