This topic was suggested by one of our users, Bart Silverstrim. He was curious about how companies could maintain an environment where maintenance downtimes would not actually affect customer experience. Think of large sites like Google, Facebook, CNN, pretty much any site in the quantcast or alexa top 100. These sites maintain extremely good uptime numbers, but how do they do it?

It’s no secret that sites like this employ a lot of servers to handle content delivery. There are not only load balanced server clusters but also CDNs and caching proxies that help mitigate some of the load on the environment. Eventually, though, all of these machines are going to need maintenance at some point. How does one do this and not affect perceived uptime from the users? This question is actually rather complex because it depends on multiple ecosystems to properly execute.

Monolithic Deployment vs. Continuous Deployment

For many years, the method of software development followed a pretty static and time consuming process:

  1. Define scope for this release
  2. Program features for the release
  3. Debug, if errors, back to #2
  4. QA (generally someone testing who ISNT the developer), if errors, back to #2
  5. Acceptance/Staging
  6. Deployment

Now, I know a lot of shops that would laugh at this list and claim that they do only #2 and #3, then shove it up to the servers. This can very well be the case, but I’d hazard a guess that those companies have never gotten close to scraping the alexa top 100, nor would I believe with a straight face if they told me they had uptimes in excess of 99%. There is another way, though: Continuous Deployment.

Simply put, Continuous Deployment is when instead of having large, multi-bug/multi-feature releases, you instead implement many, many small changes continuously throughout the product lifetime. At first glance this method might sound dangerous to many sysadmins who like to plan a task to an atomic point algorithmically, but it’s actually extremely safe if you follow some standard methodologies, namely having an environment where you can test changes (for Stack Exchange, this is meta.stackoverflow.com) and use a build proctor like CruiseControl or TeamCity to facilitate the build process.

One of the main benefits of Continous Deployment is the fact that small changes are easy to deploy and usually just as easy to revert should problems arise. It becomes significantly harder to revert a deployment if there is a large corpus of changes contained in the update.

Code it like you’ll need to change it

One thing that new programmers often fall prey to is the laziness of not properly architecting their application. I am sure that every programmer remembers their earlier code shenanigans when they’ve needed to update something they wrote earlier in their career only to find that it was fraught with all sorts of freshman mistakes. To put it simply: you need to write your code to be portable, easily understood, and utilizing the best practices available for that language. If the language is OO, this means obeying standards like MVC, creating class interfaces and making the app as dynamic as possible. The reason should be clear: code that’s easier to interface with is easier to change and update, and it generally means that bugs introduced in most modules should not have global impact to the site.

Load Balancing is Good

Load Balancing is paramount to a seamless customer experience. It allows you to do some pretty cool tricks, but only so long as your web application is coded to properly handle a load balanced environment. The main thing one needs to think about when coding an app in a load balanced environment is that there’s no guarantee that request ‘n+1’ is going to land on the same server as request ‘n’, so you need to handle sessions in a centralized/db manner so that the cookies in the browser link you up to the proper session regardless of what server you hit. This does NOT mean, however, that you should disable persistance or affinity in your load balancer! There are benefits to keeping a session on the same server with regards to file caching and the like; we just want to make sure the app is ready should you want to take one of the servers down for maintenance.

Once properly load balanced, you gain several levels of win. First off, your app will be much more performant if there are more servers available to handle the load. Secondly, if you have multiple servers available, you can bring one of them offline and your users should never know the difference. A side benefit of Load Balancing is the peace of mind you’ll get knowing that if a box eats itself alive at 3am, your site will survive without you needing to fire up the laptop or, heaven forbid, head into the office at an extremely early hour.

How Stack Exchange does it

It may become clear after reading this that what I’m talking about isn’t necessarily how Stack Exchange works. People who visit us often know that we do have site downtimes for various reasons. For those curious, the below section is how we handle our deployment and development process.

There is one big place where the Stack Exchange Core Q&A Services are vulnerable, and this is at the database. Currently we employ SQL 2008 R2 database servers (currently SP1 as of this writing), with the primary server constantly replicating to the backup server via transaction log shipping. Those familiar with transaction log shipping will know that employing this method is really only usable in an active/”hot passive” mode. One of the downsides of using SQL Server is the lack of solid high availability given our transaction load. Simply put, both the asynchronous and synchronous methods of active/active can’t keep up with the sheer volume of transactions we throw at it. We’re hoping that when Denali comes out this spring, the HA features will improve to the point where we can be fully active/active.

The reason we incur downtimes for upgrades these days has to do partially with ease of execution and cleanliness; because we use transaction log shipping, if we wanted to go active on the backup node we’d have to break the replication and convert the backup server to the master server, then re-setup replication. Would we do this for a couple windows patches? Absolutely not. It’s a nontrivial amount of work that can incur human error and is unnecessary. We reserve this process for when we do have a major database emergency.  We’re hoping that SQL 2012’s enhanced failover capabilities will permit us to enter read-only mode for brief maintenance windows, but this will need serious testing first, once Denali (2012) is released.

Development wise, this is the continuous deployment procedure that Stack Exchange uses:

  1. A developer will get the latest HEAD from the mercurial repository.
  2. That developer makes a change, then commits that change back to the Mercurial repository.
  3. TeamCity queries Mercurial (every 60 seconds) to check for new changes. If a change is found, TeamCity builds it immediately in the development testing environment.
  4. Once the developers have given the change a test in development, the developer then deploys the change to meta.stackoverflow.com.
  5. If the dev is having a good day, meta.so’s users won’t report any bugs and after a period of time the dev will push the code to everything except stackoverflow.
  6. If the change stands up to rigorous testing on the other approximately 80 sites in our network, the change will be pushed to stackoverflow, aka “the fire hose.”

One might be curious as to why we wait till the last minute to push code changes to stackoverflow. The reason behind this has to do with the fact that stackoverflow gets several orders of magnitude more traffic than the rest of Stack Exchange combined. A case study in this procedure: when we deployed ProtoBuf v2, the change worked great everywhere else, but as soon as the change was applied to stackoverflow.com, a “cold start” bug seriously degraded users’ experience.

An important thing to note as well is that a great deal of the sites’ code changes are toggleable by configuration, so if a problem is found it can be reversed and mitigated much, much faster than it would take a developer to debug and fix the problem.  Employing this method where possible in your applications will be helpful for many reasons.

Uptime! Fsck Yeah!

For most of our readership, the problems described above might not apply to you. You may not have quite as many hits as Stack Exchange, or you might have less need for high availability and uptime. This isn’t to say that the above doesn’t apply; architecting for uptime should be in every developer’s best interests and best practices.

As always, your questions and comments are most welcome, feel free to post below.

«
»
  • Interesting article, thanks for sharing. Would be very interested in your reasons for going with a Microsoft environment (and whether or not you think that has been a good thing)

    • Anonymous

       I am not really the best person to talk to since the decision to go with the Microsoft stack was made at its inception and I’m relatively new to Stack Exchange (started in October.)  A question like that is better posited to @codinghorror:twitter who may have already written a lot on this topic on his fine blog, http://www.codinghorror.com.  If you don’t read it already, it’s quite informative and witty; I’ve been an avid reader since long before I was employed here.

  • Ranjib Dey

    What you have discussed here is Continuous Delivery, its not really continuous integration. http://www.martinfowler.com/articles/continuousIntegration.html. Traditionally continuous integration ensures the health of the code base after every change (commit) by leveraging a test suite (unit/functional/integration). Continuous delivery on the other hand goes beyond that and incorporates infrastructures and deployment process automation with appropriate safety nets. 

    • sabrina D

       Yep i too had the same argument.

      • Anonymous

         I have taken your comments to heart, I’m just trying to figure out how I’d want to edit the blog post to properly differentiate between the two.  Thanks for the feedback on this.

  • Ted B

    Thanks for the informative post.  I agree with Ranjib about the distinction between Continuous Delivery and Continuous Integration.  CI does not itself determine when you push to prod — it’s concerned with how you integrate all the parts of an app before pushing to prod. I have an app environment similar to SO.  It would really help me out if you wouldn’t mind sharing what file transfer mechanism you use to push code from your build server to your Web servers.  Windows file sharing?  3rd-party SFTP server?  Thanks!

    • Anonymous

       Hi Ted, thanks for the praise!  As stated in above comments, I am cognizant of the difference and will consider how to amend the blog post to properly differentiate the two.  Regarding the copying, we use TeamCity which we have configured to use powershell calls to start a robocopy process from a windows file share.

  • Pingback: Thursday Quote: Peter Grace | John Hobbs on coding, Omaha, and life in general()