Phase Two


When the scope of the Stack Overflow engine was drastically increased, it became obvious that more hardware would be needed and that we would now have to put more emphasis on scaling.

Currently, everything lives at PEAK Internet in Corvallis, Oregon, which has worked out very well so far.  However, after much discussion, we came to the conclusion that we’ll be able to serve our globally distributed user base better if we at least move our heavier services to the East coast.  Since Fog Creek has had great experience with Peer 1, it seemed like a reasonable enough place to set up camp.  

We’ve decided to jump into two full-sized cabinets, each one with dual 40 amp power circuits pre-installed.  The current line of thinking is that we’ll spread our systems out evenly so that a serious disruption in one cabinet won’t take down service in its sister.  We want to be able to lose as many servers as possible and still be able to provide answers to you all.  How this actually works out in practice will be seen and I’m sure many revisions will follow as we learn.  

Each rack will contain two Rariton Dominion PX PDUs, one per power circuit.  These guys have a solid reputation and each unit gives us 20 receptacles for our servers: much better than the usual 8-10 receptacles that are often seen.  These units are manageable via SNMP, which allows us to monitor for spikes or other changes in activity over time.         

For basic data services, we’ll put two Dell PowerConnect 5448’s in each cabinet.  These are CLI-managed gigabit switches running a near-clone of Cisco’s IOS.  They’re great: you get basic trunking, VLAN, and SNMP support at a killer price.  Plus you can manage all of this by just ssh’ing into the switch itself.  Each switch will be interconnected with the rest (we’re protected from switching loops via STP, of course), and each server will have a connection to both switches within its parent rack.  

Now for the server hardware.  With only a few exceptions, our basic building block is the Dell R610.  These are amazingly flexible 1u boxes that are remarkably power efficient and capable of taking on almost any role assigned.   They also come loaded with the DRAC, a fantastic out-of-band management system that has saved me too many times to count.  We’ll be using these all over the place.

We’re going to run our own firewall / gateway layer on two Dell R610’s with an E5640 processor, 250GB of disk space at RAID 1, 8GB of RAM, and Intel NICs.  They’ll likely be running the OpenBSD operating system, which gives us PF for the firewall and CARP for high availability.     

There will be two load balancers built with the same hardware configuration as above, but running on top of FreeBSD.  It has been my experience that while OpenBSD is excellent for building firewalls, it hasn’t been road tested enough in other areas.  FreeBSD on the other hand, certainly has.  We’ll be using CARP again for high availability, and will run some combination of HAProxy and nginx to take care of the actual load balancing.

We’re expecting to deploy 10 web servers to start with (we currently run 6), all on the R610 platform and running Windows Server 2008 R2 Standard Edition.  They’ll have one Intel E5640, 250GB of disk space at RAID 1, and 16GB of RAM.

Two VMWare ESX servers will be built, the major difference being that these will have about a terabyte of usable storage at RAID 6 and 32GB of RAM.  

Two domain controllers will be in place, providing basic Active Directory services for all of this stuff.  They’ll be running on the 250GB of hard drive storage and 2 GB of RAM.


A syslog / monitoring server will run on a terabyte of storage at RAID 6 and 8GB of RAM.  All *nix syslog and Windows Eventlog data will be shipped over there for regular analysis and alerting if necessary.  It will be running FreeBSD.

A dedicated backup server will be in place to ensure that critical data is well protected.  We’re going to put this on an R610 with a 1.5TB of disk space and 16GB of RAM.  We’re mostly concerned about database backups, and this should give us considerable amounts of onsite storage.

The exceptions to the R610 rule are the database servers.  We’re going to put two in place, each on a Dell R710.  This shares the same guts as the R610, but is a 2u model and can house eight 2.5″ disks as compared to the six that the R610 can.  We’re going to load these up 64GB of RAM, a terabyte of storage at RAID 10, and are going to put two E5680 processors on board.

It should be understood that the above is not carved in stone: much of this will change during the actual deployment.  It’s great to lay out these big plans – they help light the way and keep you moving – but we’re not bound to any of it.  I tend to implement things organically, trying to build things in context of their environment.  What works, works. 

We’ll be posting more as things develop.