With all my talk of doing things in a scalable way comes a requirement, and that is that we actually accomplish this stuff in practice at Stack Exchange. We have been making a lot of progress in this area. George has been refining and expanding our deployment process. He improved our Windows deployment to include most of the software we use and has made it so kickstart for CentOS/Linux installs are integrated into our deployment as well.
Scaling your ability to manage your environment in my mind means doing more with less. I find I really only have to ask myself one question to quickly gauge if we are doing it right or not.
Do I have to repeat steps to do this task on multiple servers?
I like this question because it is more specific than “Is it automated?” while still implying automation. It is akin to the DRY rule in programming: “Don’t repeat yourself.”
When it comes to our environment, here is where we are at. For this I will ignore some details — for instance, we log into servers to kick off a PXE install and then just let it go — I consider that a Yes to “No repetition required” since it is only one or two steps and we don’t really deploy more than a server at a time. We don’t want to automate to the point over engineering beyond the projected size of our environment.
|Task||No repetition required||Solution or Proposed Solution|
|Windows OS Deployment||Yes||Microsoft Deployment Toolkit with LiteTouch|
|Linux OS Deployment||Yes||WDS which redirects to PXELinux and Kickstart|
|Windows OS Updates||Yes||Windows Update Services|
|Linux OS Updates||No||Kick them off with Cron or Puppet? Spacewalk?|
|Windows Firmware Updates||No||Exploring what Dell has to offer that might tie into System Center|
|Linux Firmware Updates||No||Run binaries with Puppet? (kind of scary)|
|Windows Software Deployment and Updates||No1||Microsoft System Center|
|Linux Software Deployment Updates||Yes||Puppet†|
|Windows Configuration Management||No (with the exception of IIS and web related configs)||Microsoft System Center|
|Linux Configuration Management||Yes||Puppet†|
|Automated Deployment of Monitoring and Backup Configuration||No||No ideas at this point|
1We do have some software that can be deployed via GPO, and LiteTouch deploys a lot of stack on the web servers during deployment. But future software updates and software that doesn’t lend itself to GPO is manual.
† I am currently in the middle of rolling out puppet so it is partially deployed on some of our Linux servers
The big picture of all of this is deployment as phase 1 and maintenance as phase 2 for both Windows and Linux. Also, ideally these phases connect to each other seamlessly.
One of our main goals is to change all of the above “No” to “Yes” over the next few months and then refine each step. For the most point we have deployment taken care of for both Windows and Linux. As far as maintenance goes, I believe as I progress in rolling out Puppet that will solve the vast majority of our Linux needs. How we will manage our firmware is still an unknown. As far as Window maintenance goes we are going to start exploring System Center soon and hope that it can meet our needs.
What I really think all of this will buy us is consistency, recoverability, and most importantly — time. By having all of this centrally managed it will make our servers more consitent with each other — and make them effectively drones. By having these processes automated we will be able to recover fast and replace servers easily. Lastly, and most importantly it buys time. By making our management faster and more agile, George and I can focus on rapidly deploying improvements to our environment instead of just maintaining it. By having less friction to deploying changes to our infrastructure I believe more possibilities for improvement will start emerge.