I will be upgrading the Stack Exchange network’s switching infrastructure on Saturday February, 26th. There should be minimal downtime while I do this, however the sites may be a little slow as I work through the Web Tier. Additionally there will be about a 10-20 minute complete site downtime while I move the DB servers and routers.
What exactly will I be doing? Why, I’ll be removing our current Dell PowerConnect 5448 switches and replacing them with Cisco 2960-S-48TS-L switches.
The plan is to start working on moving the web tier Saturday afternoon and be finished moving all services by early evening. I will post to Blog.Serverfault about one hour before the start of work.
I thought as a break form the normal prose some of our readers might enjoy a short overview of the Stack Exchange Network (including Stack Overflow, Server Fault, and Super User) from a technical view:
- 95 Million Page Views a Month
- 800 HTTP requests a second
- 180 DNS requests a second
- 55 Megabits per second
- 1 Rack with Peak Internet in OR (Hosts our chat and Data Explorer)
- 2 Racks with Peer 1 in NY (Hosts the rest of the Stack Exchange Network)
- 12 Web Servers (Windows Server 2008 R2)
- 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
- 2 Load Balancers (Ubuntu Server and HAProxy)
- 2 Caching Servers (Redis on CentOS)
- 1 Router / Firewall (Ubuntu Server)
- 3 DNS Servers (Bind on CentOS)
Software and Technologies Used:
- C# / .NET
- Windows Server 2008 R2
- SQL Server 2008 R2
- Ubuntu Server
- HAProxy for load balancing
- Redis for caching
- CruiseControl.NET for builds
- Lucene.NET for search
- Bacula for backups
- Nagios (with n2rrd and drraw plugins) for monitoring
- Splunk for logs
- SQL Monitor from Red Gate for SQL Server monitoring
- Mercurial / Kiln for source control
- Bind for DNS
Developers and System Administrators:
- 14 Developers
- 2 System Administrators
*(excludes fail over and management servers)
We’ve been trying to figure out what to do about the Disk subsystem IO problems we have. We spent many hours talking to vendors, trusted advisers (ok, reaaaally smart people who were willing to help us – and put up with our questions), and each other trying to find the best solution. We think we have, for now.
Before I get into what we decided to do I want to talk a little about our thought process and the requirements we put together to shape our decision.
It should be fast
I know this may seem silly but we pride ourselves on the fact that our sites load not just fast but really fast. We needed a solution that didn’t just get us to the point where our storage subsystem was good, but decimated that barrier and made it great. The added benefit to this approach is we should be able to handle the IO needs of our sites for at least the next year or so. (Assuming our growth rate projections are accurate, of course.)
It should be reasonably priced
The primary requirement of this upgrade was to get more speed out of our storage subsystem, so it makes no sense to spend $100k or more on a SAN with a whole bunch of features that we won’t use in the next 6-12 months. But we don’t want to spend the least amount of money we possibly can on equipment, either. Like Goldilocks, we needed to find a solution that was just right.
It should be reasonably safe
By safety, I mean our data won’t go up in smoke when there is a catastrophic failure — and it needs to keep our services running briskly without a ton of overhead, either mental or physical. We’re not a bank, and we don’t need to over-build our systems to never lose a single bit and always be 100% accurate. But we also respect that you put a lot of effort into making this place great, and we owe it to you to treat the content you’ve contributed with care.
Putting aside all the cool stuff, all the other feature options, does spending X on storage technology Y give us the best price/performance?
The first thing we did is to explore the different classes of server storage, and look at the pros and cons of each.
1. Direct Attached Storage enclosures
Cons: Normally can only be connected to a low number of hosts (1-4), for what you get, it’s a large investment.
2. SAN technologies
Cons: Very expensive. And to get the best performance we would need an infrastructure upgrade as well, either 10 Gigabit Ethernet or Fibre Channel.
3. PCI bus based flash drives (FusionIO)
Cons: relatively new tech, giant single point of failure with no great way to compensate for it.
4. Traditional Solid State Hard Drive (SSD) storage
Cons: Have to get non-vendor “approved” drives, since our vendor wanted insane amounts of money for a single drive.
We had grandiose plans to bring a few of the finalist options in house to do a barrage of tests, but this turned out to be harder than we thought. Storage vendors were really reluctant to allow us to bring in demo units of their hardware. When I’m looking to spend tens of thousands of dollars, I’m sorry, but I really want to either a) go to your lab and actually run test against a unit that I’m going to be buying, or b) bring a unit in house for a few weeks or a month and be able to verify that the unit will do what you claim it will. What actually happened was that we only brought in one option: the Intel X25-E solid state hard drive. The price wasn’t too expensive to bring them in on a flier, and if they didn’t work out we would be able to re-purpose them somewhere else so it wouldn’t be a total loss.
We decided to use Brent Ozar’s benchmark that he did on the FusionIO drives as our benchmark. Brent isn’t some random blogger, you understand — he is a database ninja. When he does a benchmark, he does a benchmark.
Here’s a quick comparison of our results and his:
Random Reads — 2 threads, 8 outstanding requests, 64k blocks
Random Writes — 2 threads, 1 outstanding request, 64k blocks
The numbers for the X-25′s are actually limited by the controller since the H700 has a max throughput of 600MBps. We do not know how much faster this setup could go if we put a higher throughput controller in the boxes.
We ended up not bringing in any of the SANs due to the amazing performance we got out of the Intel X25 solid state drives. Speed was all we were really after. We don’t seen the need for a SAN at this point so we’ll let that money continue to earn interest in the bank for now.
The full test that we ended up running was straight out of – you guessed it – Brent Ozar’s playbook. That us, using SQLIO to run a whole barrage of tests against the storage system.
The test system consisted of:
- Dell R710 with 96GB of memory
- 2 Xeon X5680 CPUs
- 6 Intel X25-E drives in a RAID 10 array
- H700 RAID controller with 1GB of memory
The gold standard for fast is the FusionIO drives. In our benchmarking, a RAID 10 array of Intel X25 drives got within about 25% of FusionIO performance for reads, and within 10% for writes. That exceeded everyone’s expectations and made our decision very easy. Six X25-E’s are about half the list price of one FusionIO, and putting them in a RAID array eases any concerns about reliability.
For all who are interested the raw data can be found in this archive. The archive also includes a pretty ugly Perl script that converts the raw output from SQLIO into a csv file.
> I would like to take a moment to thank all the people who helped us out looking at all these options. > > – Our vendors who put up with insane questions and flip floppery – Dell, CDW, and Fusion-IO. > – All of the regulars in the Server Fault chat room. > – Of course Brent Ozar who … is … godlike? Hmm, let’s just go with “great guy”.