I’ve recently been looking back on what we have written about our architecture in the past, and came to a stunning realization. That realization is that while we have many many different articles about what we have been doing there hasn’t been a good, solid overview of our architecture in a long time. In fact, the last really comprehensive write-up was done by Jeff before this blog even existed. And, boy I do have to say there has been quite a lot of change behind the scenes since then. So, my dear readers I’m going to take some time – and my next few blog posts – to give everyone an in depth look into how we have the Stack Exchange Network setup to serve between 12 and 14 Million page views per day.

How these posts will breakdown

Since we have obviously grown, and are offering more services to our users I’m going to break these posts out by each of the 4 major services we offer to our user base:

  • Core Q&A (this includes the API)
  • Careers
  • Chat
  • Community Blogs

Each one of these systems all work towards our goal of making the internet better, but they have different requirements and different challenges.

In this first post, I’ll be focusing on our core Q&A system, since that is after all our bread and butter.

Core Q&A

First, a high level overview of how everything is put together:

The Hardware

Our core hardware setup hasn’t changed all that much. Well, I should say the chassis haven’t changed that much. We’ve done a lot of work to upgrade the internals of the servers when needed to address performance issues as they came up, as well as handle issues that resulted from Stack Overflow being so big.

Web Tier

Of these 10 Servers, 3 are dedicated to Stack Overflow with an additional 3 servers serving Stack Overflow and the Stack Exchange Network. We have one server dedicated to Dev/QA – which also hosts meta.stackoverflow.com. Our Web Tier machines normally operate between 5 and 20% utilization. We have plenty of room to grow on these boxes.

  • 10 Dell R610 IIS web servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
    • 16 GB RAM
    • Windows Server 2008 R2
    • 2 drives
      • RAID 1
      • 2x Intel 320 300GB SSD (RAID 1)

DB Tier

We have two database server pairs. One pair is dedicated to running Stack Overflow, and the other runs the rest of the network. We run development against the secondary server of the non-stack overflow database pair. Both of our database pairs run at about 20% utilization, so once again we have room to grow here as well.

  • 2 Dell R710 database servers:
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 spindles
      • Mirrored Pair for OS
      • 6 disk RAID10 for databases
    • SQL Server 2008 R2 SP1
  • 2 Dell R710 database servers (Stack Overflow Dedicated):
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 96 GB RAM
    • 8 drives
      • Mirrored Pair for OS
      • 6 drive RAID10 of Intel X25-E SSDs for Database
    • SQL Server 2008 R2 SP1

Caching Tier

We run redundant Redis servers for our caching tier.

  • 2 Dell R610 Redis servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz
    • 16 GB RAM
    • CentOS

Network Layer

We use HAProxy for our load balancing, and Cisco Switching.

  • 2 Dell R610 HAProxy servers:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 4 GB RAM
    • Ubuntu Server
  • 6 WS-C2960S-48TS-L Gigabit switches
    • FlexStack (two stacks, 4 switches and 2 switches)

Data Integrity

As with any system, making sure that your data is backed up and the backups are good is an integral part to your service offering. We backup our databases nightly and restore them to two different locations. One local to our NY data center for our devs to work against, and one remote in our OR data center.

Conclusion

Overall I believe that we are in a good place and have plenty of room to grow given our current setup. As always we will constantly be looking at our infrastructure and tweaking it to get the best performance possible and give our users the best experience possible.

«
»
  • Peter

    How big is the SO database (full backup on disk) and how much RAM is used by the servers that host the SO db? Thanks.

    • http://blog.stackoverflow.com George Beech

      The backup is ~30GB on disk, and our memory utilization on the SO Servers is ~95GB

    • http://twitter.com/canadiancreed Chris Reed

      I remember reading somewhere that the dev DB is somewhere around the line of 20GB’s or so? Might be remembering incorrectly but it’s definitely not small.

    • http://www.facebook.com/osewa Oluwaseun Osewa

      Answer to second question: 96GB!

  • MrTomahawk

    I noticed there was no mention of virtual machines or hypervisors… Are you guys not using VMware/Hyper-V/Xen/etc, if so how do have this setup.. If not why haven’t you guys doing any type of virtualization?

    • http://blog.stackoverflow.com George Beech

      We do not use virtualization for our core infrastructure. It doesn’t really make sense for our use case in this instance. We do use VMware fairly extensively in other areas of our network that I’ll touch on in later blog posts

      • http://michielvoo.net Michiel van Oosterhout

        How do you provision a new server?

        • http://blog.stackoverflow.com George Beech

          We use WDS, I have a half written blog post about our setup that should go up at some point

  • TomV

    Out of interest, what percentage uptime has stack overflow had in it’s life?

    • http://blog.stackoverflow.com George Beech

      I can’t speak to lifetime, but we are at about 99.9% up time YTD. Which includes a couple multi-hour infrastructure upgrade down times.

  • jared

    I’m happy to not hear anything about “The Cloud”. Turns out well managed hardware is a good thing!

  • Murali Suriar

    Out of interest, why CentOS for the REDIS boxes and Ubuntu for the HAProxy boxes, rather than standardising on one Linux distribution?

    • http://blog.stackoverflow.com George Beech

      We Decided to standardize on CentOS for production servers after they where already built and rebuilding those boxes just for the fact of standardization is … silly. In the future we will use CentOS for and core functions and Ubuntu for servers where the larger package repos are useful – things like management servers and utility servers.

      • Tomasz Napierała

        Are you not concerned with CentOS problems with providing security updates?

  • Deric

    I’m suprised your Redis (caching) servers don’t have more RAM.

    • http://blog.stackoverflow.com George Beech

      We currently are only utilizing ~8GB of ram in REDIS, obviously as we grow we can add RAM to those boxes, the max on the R610′s is 128GB IIRC

  • http://www.facebook.com/hsparikh Harshil Parikh

    Hi George, will you be covering the software side of SE as well? 

  • Johannes Hansen

    I’m very interested in how you do configuration management of the linux boxes (and in general on your web servers). I’ve recently configured a pair of NginX/HAProxy servers for our ssl + load balancing needs but I have yet to settle on a good way to transfer configuration between the servers. Furthermore our failover setup is an active/passive setup using Heartbeat to switch IPs if server1 goes down, however I don’t really like this. Are your HAProxy failover setup an active/active one, or an active/passive one?

    • Anonymous

      Sounds like you already have something like Heartbeat (http://www.linux-ha.org/wiki/Main_Page) setup on your HAProxy boxes.  If you used round-robin DNS you’d have active/active load balancing. 

      • Johannes Hansen

        Thanks for the suggestion, and yes you are correct, we have Heartbeat configured. :) The DNS load-balancer solution seems very brittle to me. I mean, what happens if the server on the first IP disappears, how does the users who resolved that IP get routed to the second server? It seems like I would still need heartbeat in this setup. Maybe I’m missing something?

        • Anonymous

          You can get DNS round robin load balancing while still having failover by using two Vritual IPs. When both LBs are up, each of the LBs have one of the two Virtual IPs. If one fails, then the other will pick up its virtual IP and just hold both of them.

          • Anonymous

            Thanks for the answer… This seems a little over-the-top for my needs. Furthermore, we are getting away from the issue in my original question, which wasn’t about how to do load-balancing, but about how you share service configuration between linux servers. And secondly what kind of setup stackexchange is using… active/active or active/passive?

  • http://mikerg87.myopenid.com/ Mike J.

    i noticed you mentioned 20% utilization for most of the servers to account for growth, but with 10 already that seems like a lot of excess capacity. What utilization are you targetting before you add more hardware ? Are you going to 50% on all servers and then you will add servers 11 and 12 or is there something else you had in mind ?

  • Mxx

    Can you post pictures of your actual physical setup? Would be interesting see how you setup cable management and other wiring. Also since you designed with room to grow, how did you physically layout your servers?

  • Stefan

    What about a search service? Are you just using FULLTEXT searches in SQL? Although, I remember reading somewhere that you guys use Lucene.NET. Maybe search is implemented as part of the web service? Although you’d have to have pretty small indexes to pull that off nicely.

    • Nick Craver

      We’re using Lucene.Net on each web server, our indexes total around 5.5 GB at the moment.  The Q&A app pool itself does the indexing/searching on each web server, so 1-10 all have the same index maintained redundantly.  FULLTEXT in SQL is only used for tag-based searches in the OR case now, all other columns have been removed from the catalog.

  • Billy Reisinger

    How did you guys implement the Redis failover?

  • William Hilsum

    I read the Blog every so often and this is the first infrastructure report I have seen in a while. I am Just a little bit curious when you went all Dell and what the reasons were behind it? I have been holding off, but, I am thinking of buying in a few Dells soon and would like to hear reasons from others!

    • Anonymous

      Why Dell? The choice was made before George or I were hired, but we have stuck with it. In short, I think all vendors can be a pain but we are familiar with Dell. With Dell (vs homebuilt servers) we can get firmware update and monitoring utilities such as Open Manage.

      AD Yes they are all on AD, and the office is under the same domain. 

      • http://twitter.com/Kiril_Varbanov Kiril Varbanov

        Well, it’s not a bad choice either. I’ve had nothing but pleasure working with some real R710 machines in the past, the DRAC utility is really helpful, and the machines delivered performance like no other hardware on sale.

  • Peter Gervais

    Just wondering how you implement HA at the database layer.  Are you using mirrors or clusters?  Are they active/active?

  • http://bsilver.myopenid.com/ Bart Silverstrim

    What procedure are you using for backing up and restoring the databases?