When the scope of the Stack Overflow engine was drastically increased, it became obvious that more hardware would be needed and that we would now have to put more emphasis on scaling.

Currently, everything lives at PEAK Internet in Corvallis, Oregon, which has worked out very well so far.  However, after much discussion, we came to the conclusion that we’ll be able to serve our globally distributed user base better if we at least move our heavier services to the East coast.  Since Fog Creek has had great experience with Peer 1, it seemed like a reasonable enough place to set up camp.  

We’ve decided to jump into two full-sized cabinets, each one with dual 40 amp power circuits pre-installed.  The current line of thinking is that we’ll spread our systems out evenly so that a serious disruption in one cabinet won’t take down service in its sister.  We want to be able to lose as many servers as possible and still be able to provide answers to you all.  How this actually works out in practice will be seen and I’m sure many revisions will follow as we learn.  

Each rack will contain two Rariton Dominion PX PDUs, one per power circuit.  These guys have a solid reputation and each unit gives us 20 receptacles for our servers: much better than the usual 8-10 receptacles that are often seen.  These units are manageable via SNMP, which allows us to monitor for spikes or other changes in activity over time.         

For basic data services, we’ll put two Dell PowerConnect 5448’s in each cabinet.  These are CLI-managed gigabit switches running a near-clone of Cisco’s IOS.  They’re great: you get basic trunking, VLAN, and SNMP support at a killer price.  Plus you can manage all of this by just ssh’ing into the switch itself.  Each switch will be interconnected with the rest (we’re protected from switching loops via STP, of course), and each server will have a connection to both switches within its parent rack.  

Now for the server hardware.  With only a few exceptions, our basic building block is the Dell R610.  These are amazingly flexible 1u boxes that are remarkably power efficient and capable of taking on almost any role assigned.   They also come loaded with the DRAC, a fantastic out-of-band management system that has saved me too many times to count.  We’ll be using these all over the place.

We’re going to run our own firewall / gateway layer on two Dell R610’s with an E5640 processor, 250GB of disk space at RAID 1, 8GB of RAM, and Intel NICs.  They’ll likely be running the OpenBSD operating system, which gives us PF for the firewall and CARP for high availability.     

There will be two load balancers built with the same hardware configuration as above, but running on top of FreeBSD.  It has been my experience that while OpenBSD is excellent for building firewalls, it hasn’t been road tested enough in other areas.  FreeBSD on the other hand, certainly has.  We’ll be using CARP again for high availability, and will run some combination of HAProxy and nginx to take care of the actual load balancing.

We’re expecting to deploy 10 web servers to start with (we currently run 6), all on the R610 platform and running Windows Server 2008 R2 Standard Edition.  They’ll have one Intel E5640, 250GB of disk space at RAID 1, and 16GB of RAM.

Two VMWare ESX servers will be built, the major difference being that these will have about a terabyte of usable storage at RAID 6 and 32GB of RAM.  

Two domain controllers will be in place, providing basic Active Directory services for all of this stuff.  They’ll be running on the 250GB of hard drive storage and 2 GB of RAM.

  

A syslog / monitoring server will run on a terabyte of storage at RAID 6 and 8GB of RAM.  All *nix syslog and Windows Eventlog data will be shipped over there for regular analysis and alerting if necessary.  It will be running FreeBSD.

A dedicated backup server will be in place to ensure that critical data is well protected.  We’re going to put this on an R610 with a 1.5TB of disk space and 16GB of RAM.  We’re mostly concerned about database backups, and this should give us considerable amounts of onsite storage.

The exceptions to the R610 rule are the database servers.  We’re going to put two in place, each on a Dell R710.  This shares the same guts as the R610, but is a 2u model and can house eight 2.5″ disks as compared to the six that the R610 can.  We’re going to load these up 64GB of RAM, a terabyte of storage at RAID 10, and are going to put two E5680 processors on board.

It should be understood that the above is not carved in stone: much of this will change during the actual deployment.  It’s great to lay out these big plans – they help light the way and keep you moving – but we’re not bound to any of it.  I tend to implement things organically, trying to build things in context of their environment.  What works, works. 

We’ll be posting more as things develop.

I began working with the Stack Overflow team at the beginning of May, and one of the first challenges I faced was to find a method to stop abusive users and bots from hammering the site.  While we’re happy to make way for legitimate search engines and users of our API, we need to ensure that the majority of our cpu cycles are going towards helping everyone find answers to their questions.  We needed a rate-limiting mechanism.  

After discovering that our tried-and-true load balancer did not have a native facility for rate limiting, I turned my attention towards nginx.  A quick scan of the documentation revealed that the limit_req_zone declaration should take care of things.  After a few tests with the development team, it was determined that it might actually work.  When any user surpassed the configured rate limit, a custom 503 error page would then be displayed.

Since HAProxy is already an important part of our stack, it didn’t make sense to rip it out.  Instead, I opted to slide nginx in front of haproxy, ultimately allowing us to add rate limiting to Stack Overflow with minimal effort and less risk to everyone.  If worse came to worse, nginx could easily be pulled back out of the stack without too much interruption.  I chose to break the work up into two phases: nginx without load balancing, and nginx with load balancing. 

I made the initial deployment on Sunday, May 16, and as you can see here, there were a few small glitches as our Monday morning traffic began to flow in.  Thankfully, Geoff’s quick thinking helped me see that we had exceeded the maximum amount of connections that the Linux kernel was configured to track, and I was able to adjust things appropriately.  After that, we had a few small bouts of slowness, which were overcome simply by increasing nginx’s worker process count and reloading the config.  I left this in place for a week, making extra sure that the new layer wasn’t going to have any adverse effects.

Rate limiting was then officially enabled on Sunday, May 23.  There was a bit of shakiness at first, as I did not enable the ‘nodelay’ feature.  This caused nginx to actually try and slow down offenders instead of just handing them 503s.  Once this was discovered, the feature was turned on and we were in business.

Over the past few days, I’ve been pouring over our http access logs, trying to determine if we have seen any amount of success.  I’m happy to say that we have: where I normally would see dozens of bots hitting us somewhere between 16,000 and 40,000 times an hour, I saw… none.  Victory for nginx! 

More to follow as we continue to tweak nginx and employ other systems to help us keep the channels clean…