Our Storage Decision

George Beech

We’ve been trying to figure out what to do about the Disk subsystem IO problems we have. We spent many hours talking to vendors, trusted advisers (ok, reaaaally smart people who were willing to help us – and put up with our questions), and each other trying to find the best solution. We think we have, for now.

Before I get into what we decided to do I want to talk a little about our thought process and the requirements we put together to shape our decision.

Requirements

It should be fast

I know this may seem silly but we pride ourselves on the fact that our sites load not just fast but really fast. We needed a solution that didn’t just get us to the point where our storage subsystem was good, but decimated that barrier and made it great. The added benefit to this approach is we should be able to handle the IO needs of our sites for at least the next year or so. (Assuming our growth rate projections are accurate, of course.)

It should be reasonably priced

The primary requirement of this upgrade was to get more speed out of our storage subsystem, so it makes no sense to spend $100k or more on a SAN with a whole bunch of features that we won’t use in the next 6-12 months. But we don’t want to spend the least amount of money we possibly can on equipment, either. Like Goldilocks, we needed to find a solution that was just right.

It should be reasonably safe

By safety, I mean our data won’t go up in smoke when there is a catastrophic failure — and it needs to keep our services running briskly without a ton of overhead, either mental or physical.  We’re not a bank, and we don’t need to over-build our systems to never lose a single bit and always be 100% accurate. But we also respect that you put a lot of effort into making this place great, and we owe it to you to treat the content you’ve contributed with care.

Putting aside all the cool stuff, all the other feature options, does spending X on storage technology Y give us the best price/performance?

Research

The first thing we did is to explore the different classes of server storage, and look at the pros and cons of each.

1. Direct Attached Storage enclosures

Pros: Fairly low cost, although more expensive than internal storage. You get a large number of spindles to run faster with.
Cons: Normally can only be connected to a low number of hosts (1-4), for what you get, it’s a large investment.

2. SAN technologies

Pros: Flexible storage, generally expandable (for a price), lots of disks.
Cons: Very expensive. And to get the best performance we would need an infrastructure upgrade as well, either 10 Gigabit Ethernet or Fibre Channel.

3. PCI bus based flash drives (FusionIO)

Pros: ridiculously fast. not a horrible price. Did i mention F-A-S-T FAST?
Cons: relatively new tech, giant single point of failure with no great way to compensate for it.

4. Traditional Solid State Hard Drive (SSD) storage

Pros: Fast, potentially really fast. flexibility for future upgrades without sinking too much money into one technology.
Cons: Have to get non-vendor “approved” drives, since our vendor wanted insane amounts of money for a single drive.

Evaluation

We had grandiose plans to bring a few of the finalist options in house to do a barrage of tests, but this turned out to be harder than we thought. Storage vendors were really reluctant to allow us to bring in demo units of their hardware. When I’m looking to spend tens of thousands of dollars, I’m sorry, but I really want to either a) go to your lab and actually run test against a unit that I’m going to be buying, or b) bring a unit in house for a few weeks or a month and be able to verify that the unit will do what you claim it will. What actually happened was that we only brought in one option: the Intel X25-E solid state hard drive. The price wasn’t too expensive to bring them in on a flier, and if they didn’t work out we would be able to re-purpose them somewhere else so it wouldn’t be a total loss.

We decided to use Brent Ozar’s benchmark that he did on the FusionIO drives as our benchmark. Brent isn’t some random blogger, you understand — he is a database ninja. When he does a benchmark, he does a benchmark.

Here’s a quick comparison of our results and his:

Random Reads — 2 threads, 8 outstanding requests, 64k blocks

FusionIO Intel X25-E
in RAID10
MB/s 1424 1064
IOs/s 22788 17023

Random Writes — 2 threads, 1 outstanding request, 64k blocks

FusionIO Intel X25-E
in RAID10
MB/s 632 584
IOs/s 10114 9337

The numbers for the X-25′s are actually limited by the controller since the H700 has a max throughput of 600MBps. We do not know how much faster this setup could go if we put a higher throughput controller in the boxes.

We ended up not bringing in any of the SANs due to the amazing performance we got out of the Intel X25 solid state drives. Speed was all we were really after. We don’t seen the need for a SAN at this point so we’ll let that money continue to earn interest in the bank for now.

The full test that we ended up running was straight out of – you guessed it – Brent Ozar’s playbook. That us, using SQLIO to run a whole barrage of tests against the storage system.

The test system consisted of:

  • Dell R710 with 96GB of memory
  • 2 Xeon X5680 CPUs
  • 6 Intel X25-E drives in a RAID 10 array
  • H700 RAID controller with 1GB of memory

The gold standard for fast is the FusionIO drives. In our benchmarking, a RAID 10 array of Intel X25 drives got within about 25% of FusionIO performance for reads, and within 10% for writes. That exceeded everyone’s expectations and made our decision very easy. Six X25-E’s are about half the list price of one FusionIO, and putting them in a RAID array eases any concerns about reliability.

For all who are interested the raw data can be found in this archive. The archive also includes a pretty ugly Perl script that converts the raw output from SQLIO into a csv file.

> I would like to take a moment to thank all the people who helped us out looking at all these options. > > – Our vendors who put up with insane questions and flip floppery – Dell, CDW, and Fusion-IO. > – All of the regulars in the Server Fault chat room. > – Of course Brent Ozar who … is … godlike? Hmm, let’s just go with “great guy”.

«
»
  1. Tweets that mention Our Storage Decision - blog.serverfault.com -- Topsy.com says:

    [...] This post was mentioned on Twitter by Joel Spolsky, Stefan Plattner, Hubert Taler, nate.vaughn, Michael Contento and others. Michael Contento said: RT @GABeech: How did we make the #stackoverflow disk system faster? http://goo.gl/9epp5 #sysadmins #storage [...]

  2. Jeff Blaine says:

    For those like me who wondered, the Intel X-25-E is a SSD

  3. Wesley Brown says:

    Be very careful. The Intel/2.5″ SATA SSD market in general has had crazy high failure rates. I’m not talking bit or bytes bad whole disks just going south. I’m not Brent but I do some benchmarking and work with storage a bit. I post that up at http://www.sqlserverio.com if you have any questions just drop me a line!

  4. Anonymous says:

    WOW! You’re too kind, sir. Always a pleasure working with you guys.

  5. Jeff Atwood says:

    MLC or SLC? There are MASSIVE differences in tech, so you have to be specific.. see http://serverfault.com/questions/229833/is-it-safe-to-use-consumer-mlc-ssds-in-a-server

  6. Anonymous says:

    It’s amazing what you can do without the “Enterprise” groupthink/mindset. If you pitched this at the great big enterprisy bureaucracy I work in, you’d be laughed at and given a thousand and one reasons why it could never work.

  7. Anonymous says:

    “I’m not Brent” – hahaha, you rock, sir. You ain’t got nothin’ to be ashamed of.

  8. Hans Løken says:

    Still I wonder though, won’t putting a bunch of SSD behind a RAID controller (which basically abstracts away the fact that it is an SSD from the OS) keep SSD maintenance tasks like TRIM from running, i.e. won’t it seriously reduce the expected lifetime of an SSD?

  9. Hans Løken says:

    Duplicate removed.

  10. Sysadmin Sunday #18 « Boxed Ice Blog says:

    [...] Our Storage Decision [...]

  11. Scaling up or scaling out? | Brent Ozar – Too Much Information | Brent Ozar - Too Much Information says:

    [...] Swap out the storage for solid state drives – like Intel X-25 consumer drives as StackOverflow is doing [...]

  12. Anonymous says:

    Intel has drivers that enable the usage of TRIM for SSD’s in a raid array: http://www.bit-tech.net/news/hardware/2010/03/23/intel-releases-trim-for-raid/1

  13. Anonymous says:

    So is TRIM running on the X25 RAID 10? If so, anything special needed to make it so?

  14. Jim says:

    How did you get around the clustering problem?

  15. Stack Overflow架构升级之路 says:

    [...] their IO system they selected a RAID 10 array of Intel X25 solid state drives . The RAID array eased any concerns about [...]

  16. Joe says:

    Great post! Did you guys consider the OCZ Revo drives? They seem to represent a sort of halfway mark between the FusionIO performance and X-25 pricing levels. They also feature onboard RAID capability and are available in huge capacities. http://techgage.com/article/ocz_revo_120gb_pci_express_ssd_performance_preview/ Just wondering…

  17. Ben Norris says:

    Will be interesting to see what speeds these drives give after a bit of usage. Also how quickly they die under intensive use.

  18. New Hardware for Stack Overflow Database - Server Fault Blog says:

    [...] of talking about some of the growing pains we have been having with our database server. And how we are going to deal with it. Well we will be moving Stack Overflow’s database to it’s own hardware [...]

  19. [repost] Stack Overflow Architecture Update – Now at 95 Million Page Views a Month » New IT Farmer says:

    [...] their IO system they selected a RAID 10 array of Intel X25 solid state drives . The RAID array eased any concerns about [...]

  20. Dan Cooper says:

    You say “and putting them in a RAID array eases any concerns about reliability”.

    What about garbage collection (i.e. TRIM ATA command)? Currently that i know if, no hardware RAID card supports TRIM, so as you see good performance now, that can degrade over time as your NAND cells are consumed if you aren’t performing proper garbage collection.

    As of Server 2008 R2 it is supported, but no support for RAID cards of as of yet (that i have heard of, to this date).

    I only ask this because I am looking at this kind of solution for my professional and personal business and dont want to back ourselves into a corner.

  21. Matthew Tibbits says:

    How about backBlaze??

  22. M. Tibbits says:

    How about BackBlaze

  23. Steven Baker says:

    how is performance going now?  looking at similar setup but worried over lack of trim support in the raid controller, so wondered what performance is looking like now

  24. Ryan Turner says:

    How’s this working out 10 months on? Great write up.. thanks!!

  25. Stack Overflow running short on space « Nick Craver says:

    [...] We looked at a lot of  storage options to solve that IO problem.  In the end, we went with the fastest SSDs money could buy.  The configuration on that new server is a RAID 1 for the OS (magnetic) and a RAID 10 6x Intel X-25E 64GB, giving us 177 GB of usable space.  Now let’s do the math of what’s on that new box as of today: [...]

  26. Jordan Rieger says:

    Actually the Intel drivers only pass TRIM commands to drives that are on a RAID controller and not participating in a RAID array, or only participating in a RAID 0 array. Meaning effectively no TRIM for RAID 0+1, 1, 10, etc. drives. See http://www.anandtech.com/show/6161/intel-brings-trim-to-raid0-ssd-arrays-on-7series-motherboards-we-test-it

    But as long as you keep 10~20% free space on each drive, you should be able to rely on the drive’s internal controller’s periodic idle-time garbage collection feature.

    Jordan Rieger

  27. Tracker1 says:

    When I partition my SSDs in RAID, I usually only partition 80% of the available space to ensure that buffer is left open.

  28. Sean H. Worthington says:

    How have things worked out after two years?

Leave a Reply