I thought as a break form the normal prose some of our readers might enjoy a short overview of the Stack Exchange Network (including Stack Overflow, Server Fault, and Super User) from a technical view:

Traffic:

  • 95 Million Page Views a Month
  • 800 HTTP requests a second
  • 180 DNS requests a second
  • 55 Megabits per second

Data Centers:

Production Servers*:

  • 12 Web Servers (Windows Server 2008 R2)
  • 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
  • 2 Load Balancers (Ubuntu Server and HAProxy)
  • 2 Caching Servers (Redis on CentOS)
  • 1 Router / Firewall (Ubuntu Server)
  • 3 DNS Servers (Bind on CentOS)

Software and Technologies Used:

Developers and System Administrators:

  • 14 Developers
  • 2 System Administrators

*(excludes fail over and management servers)

«
»
  1. phsr says:

    Are you going to post details on the hardware of the various machines?

  2. Kyle Brandt says:

    You can find that detail at http://blog.serverfault.com/post/1432571770/ , there have been some DB upgrades since that post however.

  3. Chamiltongt says:

    What are you guys using for version control? I know, boring, but I’m curious.

  4. Josh Kodroff says:

    What’s the Ubuntu Server for? HAProxy?

  5. Darren Newton says:

    Why Redis on CentOS and not Ubuntu like HAProxy?

  6. Kyle Brandt says:

    Oops forgot about source control :-) Updated the post, it is Mercurial (and many use the kiln interface)

  7. Matt Phillips says:

    Looks pretty cool to me!

  8. Guest says:

    Where is the “updated post”, I don’t see any mention of VCS in the above blog post. ;-)

  9. Tweets that mention Stack Exchange’s Architecture in Bullet Points - blog.serverfault.com -- Topsy.com says:

    [...] This post was mentioned on Twitter by rands, Joel Spolsky, Brent Ozar, Mike Champion, Red Gate Software and others. Red Gate Software said: RT @way0utwest: Woo-hoo, Stack Exchange uses #sqlmonitor http://twurl.nl/1d880s [...]

  10. masebase says:

    Built in MVC, not web forms, right?

  11. Faizan Javed Ph.D. says:

    Architecture still seems to be favoring scaling-up vs scaling-out, is that correct?

    Does the team use any key-value store systems for analytics processing?

  12. wdh says:

    Is there any CDN in front of things at all? I’m assuming no, but wanted to verify.

  13. Michael Haren says:

    @Kyle,

    Can you expand on the http requests (800/s) vs page views (95m/mo) figures a bit? Is 800/s a peak figure? I’m just curious because 95,000,000/(seconds in a month) is more like 36 hits/second. Is 96% of your traffic really API or Ajax calls? (It’s also possible I’m an idiot.)

  14. Scott R. Jones says:

    Are you guys still using LINQ to SQL?

  15. wil says:

    Awesome! Amazing how few servers considering the popularity/throughput – Good admin! :)

  16. Stack Exchange’s Architecture in Bullet Points « Interesting Tech says:

    [...] Read more here Posted in Uncategorized , interesting, science, tech | No Comments » [...]

  17. ahm says:

    What does backula backup in your architecture?

  18. Alex Kahn says:

    Every asset served and AJAX request is also an HTTP request. ‘Page views’ doesn’t include these types of requests.

  19. Michael Haren says:

    But really, 96% of your hits are not actual pageviews?

  20. Stack Exchange’s Architecture in Bullet Points – blog.serverfault.com « Netcrema – creme de la social news via digg + delicious + stumpleupon + reddit says:

    [...] Stack Exchange’s Architecture in Bullet Points – blog.serverfault.comblog.serverfault.com [...]

  21. render says:

    nicely done. Not into the MS stack but nice job anyway.

  22. NerdyRoom™ » Stack Overflow, Server Fault und Co says:

    [...] http://blog.serverfault.com/post/stack-exchanges-architecture-in-bullet-points/ [...]

  23. Richard says:

    Hey Michael, you might wanna take a look at this: http://www.quantcast.com/p-c1rF4kxgLUzNc Click on ‘month’ and select ‘impressions’ from the left select box

  24. Kyle Brandt says:

    Hi Michael,

    Each pageview is going to make multiple HTTP requests. These include requests to our cookie-less sstatic.net static content if that data has not been cached on the client as well things like Ajax requests and API requests. But yes, 800 req/s is the peak amount of requests (Actually a little higher). From a sysadmin perspective I am mostly interested in my peak figures as this is what we need to make sure we can handle.

  25. Kyle West says:

    I’d like to know the answer to that too!

  26. Kyle Brandt says:

    Hi Darren,

    I actually ran into a problem with Ubuntu server where it would boot into ‘runlevel unknown’ which has made me a little wary of Ubuntu server (I was able to fix it, but the problem has been around for a few releases). For my monitoring and backup systems I really favor Ubuntu as it has a package for everything and I can install new tools easily without hunting for repositories or compiling the programs. For things like the router and HAProxy I will probably opt for CentOS going forward. However, I don’t feel so strongly about this that I felt the need to rebuild the haproxy servers. That reminds me, I totally forgot to mention our 3 bind DNS servers — time to update the post!

  27. Michael Haren says:

    Thanks for clarifying, Kyle. It’s clearly an impressive system–good work!

  28. Linux Thegreat says:

    I bet those are beefy servers whose specs have been conveniently left out. Look we’re only on two servers and it only cost us $100K + hardware costs. And I’m guessing $100K is nowhere near the actual cost since you have to pay by CPU or CAL. Pooh on the open source guys, they need a bunch of $10K servers to get the same results.

  29. wil says:

    I’m guessing you have not seen statistics for some sites in similar usage to SO – You would be surprised…

  30. George Beech says:

    No we do not use a CDN. All static content is served off the sstatic.net domain hosted in our datacenters.

  31. Charlie says:

    If you like Ubuntu’s ‘package for everything’, you might like Arch Linux. It’s a little lighter than Ubuntu, and everything in their repos is up-to-date.

  32. Railmeat says:

    I am curious about how you are using Bacula for backup. Are you backing up to tape or disk to tape? I assume the actual backup server is running Linux.

  33. Marketing Related Articles | Zillion Bits says:

    [...] are the leading minds and landmark ideas that have established the Harvar… Related BlogsStack Exchange’s Architecture in Bullet Points – blog.serverfault.com Here are few related [...]

  34. Stack Exchange says:

    Yes, where it makes sense – in other places we’re using raw SQL.

  35. Nick Craver says:

    Yup, this is correct, we’re moving to MVC3 now (like right now, chat is already running on it).

  36. James says:

    RReally like this post. Generous gift out to the community.

  37. Ravelry in Bullet Points -- Code Monkey Island says:

    [...] by Stack Exchange’s Architecture in Bullet Points. I have to admit that I was happy to see that we stack up pretty well, being a Ruby on Rails site. [...]

  38. Nick says:

    Hi there, you have 2 HaProxy load balancers but then say * excludes fail over. I assume 1 at each DC? Do you have any failover for the load balancers, and how do you manage that/them (heart beat?), why not use Peer1′s ‘community’ load balancer – any issues using your own?

  39. NotNow says:

    Question: Why prefer splunk over developing your own solution?

  40. Nick Craver says:

    We’re very open about the hardware…we simply didn’t repeat it in this blog post. You can find the complete specs in an earlier post on this very blog: http://blog.serverfault.com/post/1432571770/

  41. RyanD says:

    for production, I’d rather stick with a widely accepted stable release than the newest version deemed stable by the package maintainer.

  42. Warren says:

    My guess is that it’s substantially cheaper (currently) to use an existing tool that has support than it is to roll their own

  43. Daily del.icio.us for February 12th through February 15th — Vinny Carpenter's blog says:

    [...] can take advantage of the modern web with the first round of HTML5 support within the GWT SDKStack Exchange’s Architecture in Bullet Points – blog.serverfault.com – I thought as a break form the normal prose some of our readers might enjoy a short overview [...]

  44. George Beech says:

    We do both. We backup to disk for fast retrieval and to tape for historical archiving.

  45. Matt says:

    Hi Kyle thanks for posting. Would love to know how you decided on Redis vs. the alternatives like Memcached, Microsoft’s alternative, etc?

    Also what are some of your use cases for Splunk? Love it.

  46. Matt says:

    Log analysis and event correlation is a difficult problem and massive effort to build yourself. Why not default to a pre-existing product? Splunk seems affordable.

  47. Exectweets » spolsky at 02/11/11 07:31:42 says:

    [...] Pro Tweets The current Stack Exchange architecture, in bullet points http://blog.serverfault.com/post/stack-exchanges-architecture-in-bullet-points/ spolsky – Fri 11 Feb 19:31 previous [...]

  48. Iamxjb says:

    use Asp.net mvc ?

  49. pauska says:

    -One- firewall? What happens if it breaks or gets rebooted? Have you considered dual ASA’s in stateful active/passive failover, or something like CARP atleast?

  50. OpenQuality.ru | Качество программного обеспечения says:

    [...] • Stack Exchange: статистика по проекту и список задействованных программных продуктов. [...]

  51. Todd Huss says:

    95M page views per month on only 12 app servers and 2DB server is truly impressive! Nice work!

  52. links for 2011-03-03 » krisd's blog says:

    [...] Stack Exchange’s Architecture in Bullet Points – blog.serverfault.com [...]

  53. George Beech says:

    Fine print at the bottom: *(excludes fail over and management servers)

    :)

  54. Iamxjb says:

    more info :http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html

  55. Stack Overflow架构升级之路 says:

    [...] update. A recent series of posts talks a lot about how they’ve been handling their growth: Stack Exchange’s Architecture in Bullet Points, Stack Overflow’s New York Data Center, Designing For Scalability of Management and Fault [...]

  56. Interesting Links « Endlessly Curious says:

    [...] Stack Exchange’s Architecture in Bullet Points. Architectural information about one of the biggest sites I know running a mostly Microsoft server stack. [...]

  57. [repost] Stack Overflow Architecture Update – Now at 95 Million Page Views a Month » New IT Farmer says:

    [...] update. A recent series of posts talks a lot about how they’ve been handling their growth: Stack Exchange’s Architecture in Bullet Points, Stack Overflow’s New York Data Center, Designing For Scalability of Management and Fault [...]

  58. Delicious Bookmarks for March 13th through March 14th « Lâmôlabs says:

    [...] Stack Exchange’s Architecture in Bullet Points – Server Fault Blog – March 13th ( tags: architecture stackoverflow scalability web hardware server scaling technology stackexchange ) [...]

  59. Diego Barros says:

    Are you guys using any sort of message queuing for background tasks?

    For example, sending emails, resizing images (if you needed to do this), etc. Anything that doesn’t have to be done when a web server request is made.

  60. Richard says:

    Kyle, how are you making your SQL Servers highly available? Are they setup in a cluster or are you using mirroring?

  61. SQL is dead! Long live SQL! : accidental hacker says:

    [...] per second, totaling 95 million page views per month with 2 (yes two) SQL Server machines. 1  Granted, this is nowhere near Facebook-level traffic, but I can’t help but wonder how [...]

  62. Anonymous says:

    Very good information, thanks! It is nice to see stuff that can be applied to the not google/facebook scale websites as well! 

  63. Joy George says:

    Thanks for the post..I don’t know is it right to ask..What are the best practices you are taking while calculating the badges and all? Is it a background task? or done when people click on the points link or profile page?

  64. Kaye says:

    Awesome! i’m not a system administrators. why 180 DNS requests a second needs 3 DNS server ?

  65. Deivid Martins says:

    Awesome! What is the page views a month now in 2012?

    Best,

  66. Novkovski Stevo Bato says:

    Do you use 12 physical web servers? If so, then what hardware configuration they have?

  67. sm says:

    I know it is kind of late to ask it – but could you elaborate how we would be surprised?

    Is the amount of hardware high or low compared to other sites? What is the usage scenario on these other sites: seldom updates and a relativly static page grwoth or many small updates every second (like on so)? How is the search beeing used?Thanks for any feedback

  68. Erick Machado Alvarez says:

    The application is centralized? How do servers to communicate with the application?

  69. Demetri Obenour says:

    dead link

  70. Nick Craver says:

    Thanks for the heads up – at some point that link format died. I’ve updated to the current link here: http://blog.serverfault.com/2010/10/29/1432571770/

  71. Josh says:

    Would be great if you updated this to current…

Leave a Reply