With all my talk of doing things in a scalable way comes a requirement, and that is that we actually accomplish this stuff in practice at Stack Exchange. We have been making a lot of progress in this area. George has been refining and expanding our deployment process. He improved our Windows deployment to include most of the software we use and has made it so kickstart for CentOS/Linux installs are integrated into our deployment as well.

Scaling your ability to manage your environment in my mind means doing more with less. I find I really only have to ask myself one question to quickly gauge if we are doing it right or not.

Do I have to repeat steps to do this task on multiple servers?

I like this question because it is more specific than “Is it automated?” while still implying automation. It is akin to the DRY rule in programming: “Don’t repeat yourself.”

When it comes to our environment, here is where we are at. For this I will ignore some details — for instance, we log into servers to kick off a PXE install and then just let it go — I consider that a Yes to “No repetition required” since it is only one or two steps and we don’t really deploy more than a server at a time. We don’t want to automate to the point over engineering beyond the projected size of our environment.

Task No repetition required Solution or Proposed Solution
Windows OS Deployment Yes Microsoft Deployment Toolkit with LiteTouch
Linux OS Deployment Yes WDS which redirects to PXELinux and Kickstart
Windows OS Updates Yes Windows Update Services
Linux OS Updates No Kick them off with Cron or Puppet? Spacewalk?
Windows Firmware Updates No Exploring what Dell has to offer that might tie into System Center
Linux Firmware Updates No Run binaries with Puppet? (kind of scary)
Windows Software Deployment and Updates No1 Microsoft System Center
Linux Software Deployment Updates Yes Puppet†
Windows Configuration Management No (with the exception of IIS and web related configs) Microsoft System Center
Linux Configuration Management Yes Puppet†
Automated Deployment of Monitoring and Backup Configuration No No ideas at this point

1We do have some software that can be deployed via GPO, and LiteTouch deploys a lot of stack on the web servers during deployment. But future software updates and software that doesn’t lend itself to GPO is manual.

† I am currently in the middle of rolling out puppet so it is partially deployed on some of our Linux servers

The big picture of all of this is deployment as phase 1 and maintenance as phase 2 for both Windows and Linux. Also, ideally these phases connect to each other seamlessly.

One of our main goals is to change all of the above “No” to “Yes” over the next few months and then refine each step. For the most point we have deployment taken care of for both Windows and Linux. As far as maintenance goes, I believe as I progress in rolling out Puppet that will solve the vast majority of our Linux needs. How we will manage our firmware is still an unknown. As far as Window maintenance goes we are going to start exploring System Center soon and hope that it can meet our needs.

What I really think all of this will buy us is consistency, recoverability, and most importantly — time. By having all of this centrally managed it will make our servers more consitent with each other — and make them effectively drones. By having these processes automated we will be able to recover fast and replace servers easily. Lastly, and most importantly it buys time. By making our management faster and more agile, George and I can focus on rapidly deploying improvements to our environment instead of just maintaining it. By having less friction to deploying changes to our infrastructure I believe more possibilities for improvement will start emerge.

«
»
  • kevin

    Why do you guys have such a mix?  Why not go with just Linux or just Windows?

    • Mark Henderson

      I’m guessing it’s because they need to pick the best tool for the job.

      Stack Exchange runs on Windows/IIS/SQL Server, but I don’t know of a single decent HA Load Balancer that runs on Windows. Windows has an OK variety of firewalls (TMG comes to mind), but *nix pretty much wins hands-down from a firewall point of view as well.

  • Zoltán Mezei

    For the linux upgrades (in case you use RHEL or any distribution based on it) I recommend mrepo ( http://dag.wieers.com/home-made/mrepo/ ). Spacewalk is nice, but it’s either unstable (with postgresql support) or doesn’t scale well (with Oracle XE support). Automating the upgrades can be done through puppet as well.

  • Zoltán Mezei

    As for the backup/monitoring configuration: you miss something from your table and it is some way to manage your inventory (though puppet and/or SCCM can be used for this). If you have an inventory, you can use its data to update your backup/monitoring configurations automatically.

    I do this in a pure linux environment with ocsinventory, puppet, zenoss and bacula with the help of some very simple scripts. Works like charm…

  • pjz

    The update/upgrade issues are why I prefer debian-based distributions; point them at a local repo and go.

    Easy to extend to your own software by packaging that up too.

    FAI is the debian kickstart equivalent.

    I must admit to complete ignorance re: Windows.

  • Type your comment her I’m guessing you have not seen statistics for some sites in similar usage to SO – You would be surprised…

    e.

  •   Do they have any stated performance goals?  How do they monitor
    

    site performance under load? These would seem to be important questions to ask of any site that gets profile.

  • I just got on the blogs and I am glad I discovered your blog. I will return. I would like to become familiar with learning RSS feeds. Perhaps that is what I will learn soon.

  • Great article and lots of information. It is good to read about those who have come before.