We have been live at the New York data center with stackoverflow.com, superuser.com, serverfault.com and the rest of the Stack Exchange network for a work week now and the SysAdmin team is pleased to say that it has gone well. It has been smooth since launch with the only downtime being for some quick tweaking we wanted to try.

So how does the performance look when it comes to our improved latency promise?

We use Pingdom to monitor our servers from outside our network. This data is the average time to load the Stackoverflow.com homepage from all the Pingdom locations which includes locations throughout Europe, The United States, and Canada to load the Stackoverflow.com homepage. Given, this is not exactly scientific, but the big drop in starting the date of the move (October 24th) is a big win in my book. When I, well, the Pingdom web interface, ran these numbers for the Texas and Chicago I also saw an improvement in the latency.

Diagrams Please
When we got an email of someone offering to do a network diagram for us I knew it was time to get something up:

This is just an overview of the network — on the left is our public facing network and the right side is our private network. There might be some reworking of the network layer down the road (more on that later).

The Web Servers:
We now have 4 more web servers which are also more powerful. These are all Windows Server 2008 R2.

Old Servers:

New Servers:

We now keep all the web server configs the same so adding another server for a site is just a load balancer change. So currently stackoverflow.com runs on 6 servers (3 servers dedicated for stackoverflow only).

The CPU graphs on our web servers show just how much of better performers they are for us. The following is a comparison of dedicated stackoverflow.com web servers (Keep in mind that these are on 6 servers instead of 3 now).


New: In general the CPU load on our web servers has been balanced and it still is. In short, we have plenty of room for growth here.

The Database Servers:
Our database servers are also more powerful. Old Database Servers:

New Database Servers:

We are currently working through a full text performance regression (Possibly going from SQL 2008 to R2) and will upload graphs when we have a more meaningful comparison.

The Network Layer:
Our router/firewall are both Linux machines with the same basic specs as the web Tier but with less memory. My goal has been perfect redundancy at the network layer and to eliminate all SPOFs. To this end we have two switches, CARP on our LAN and a private BGP peering with our provider, redundant switches and bonded nics. I got close but tests show that I have failed. Right now at the heart of this problem there are two things:

  1. Quagga just doesn’t work the way I would expect dynamic routing protocols to work. If a route disappears it keeps announcing it unless I redistribute static routes. Also, if a connected route goes down and it also knows about the route through iBGP it doesn’t insert the iBGP route into the table but does remove the static route.
  2. Stateful Firewalls and asymmetric routes don’t like each other. The fundamental problem is that they would need to sync state faster than SYN and then the SYN ACK or suffer performance problems. There is an interesting whitepaper on this topic: “Demystifying cluster-based fault-tolerant Firewalls.”

George and I were left with two options, try to force symmetry in the entire network or eliminate statefulness. After a day of back and forth we have decided that Linux/Quagga isn’t working for us and that we also need to separate out our router and firewall levels. Our plan is currently to move to Cisco because we are familiar with them and the time cost to explore Juniper or BSD is not appealing to us at the moment. Currently however we can lose either the router our either WAN link and still automatically come back up. So do have redundancy, it just isn’t perfect yet.

Redis and our Load Balancers:
These are also currently spec’ed out the same as our web tier machines except that the load balancers have 4 GB or RAM. They run Linux (Currently a mix of Ubuntu Server and CentOS). If you have been following this blog you know we use HAProxy and love it. We also use Redis as our distributed caching layer.

Internal Network:
Besides the database our Internal network has the following:

  • A Linux backup Server running Bacula which also handles our web logs and web log analysis (R610, 32 GB Ram, Xeon E5640)
  • A Linux monitoring Server with Nagios and Splunk and n2rrd for Graphs. (R610, 32 GB Ram, Xeon 5640)
  • 2 ESX Servers for our Domain Controllers (R610, 16 GB Ram, Xeon 5640)

From a sysadmin perspective, all that ram gives us the ultimate log analysis playground real estate.

What is next?
We still have to move our sstatic.com domain. George is currently building out our in house external facing DNS servers. We also still have our Oregon data center so some geographic load balancing of the static content is on our wishlist. Also, we are trying to gather all the stats we can into Nagios so we can measure the impact of our tweaks, trend, solve problems quickly, and enjoy the info porn.

We scale in three directions up, out, and awesome people. We don’t have lots of small weak servers (out) nor do we have a couple of monsters (up) — rather we strike a balance. Our third, and most import scaling direction is getting awesome people. By that I mean we have great programmers who happen to be performance junkies. They literally tune the code 24/7 and also know a lot about SQL and system tuning as well. The sysadmin team will be working with them to get the best damn performance we can from our new machines.