At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.

Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?

Why Use Virtualization?

Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.

This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:

  • Live migration (vMotion in VMWare terms)
  • Running multiple operating systems (i.e. Windows and Linux) on the same hardware
  • Easier to get full utilization of hardware resources by moving VMs around

These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?

The Joy of Commodity Hardware

As Wikipedia defines it:

Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”

Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.

So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.

Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.

Virtualization and Centralized Storage are Best Friends

With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.

Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.

Virtualized Clusters to the Rescue?

If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.

VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.

Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.

What Will We Go With?

Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.

  • Miles Erickson

    There is a third choice: build a SAN using commodity hardware using VMware-certified SAN software. This both eliminates vendor lock-in and enables the pursuit of far greater performance than can be achieved at a similar price point using proprietary SAN hardware. I recently had the opportunity to pursue a head-to-head comparison of proprietary vs. open SAN hardware with a previous employer, and we quickly realized that there was no comparison at all within our limited (sub-20K) budget: we could buy an anemic bottom-of-the-line proprietary box, or a high-performance ZFS storage server with SSD caching of both reads and writes. (Vendor names omitted in the interest of neutrality.)

    • zippy

      We went on a similar route for our SAN. You’re right that going with an open SAN solution gives you the best performance and space for your buck. One thing you lose though is 24/7 4hr onsite mission critical support. When your SAN goes down and everything in your org is on it, you want to be able to get someone to fix it ASAP now.

      • When you can afford 24/7 support, you get a technician flown in from anywhere and you don’t care about investment costs.

        When you can only afford/need next-business-day, you get some local schlock, who has seen the system only in the lab or comes to a linux shop and requests a windows machine with an HBA to launch his management console.

    • Mark Lawrence

      I loved this post and it has given me food for thought.  I have an iSCSI SAN that has 16 15K SAS drives, but its two controllers won’t break any records in terms of performance.  However the unit allows me to swap out the iSCSI controller modules and put in DAS modules so that the unit effectively becomes a JBOD.  As a performance upgrade I can then use a server with lots of RAM, stick on Windows Storage Server 2008 r2 and a couple of good controllers, plus a couple of 10gb HBAs to act as a head unit, and attach the JBOD to it – this effectively allows me to re-use the unit as a high performance ass-kicking 10 gb SAN. 

      Because I’m a Hy-PerV and not using VMware I can get away with this.  At the moment it’s just a pipe dream though, I’ve just got a 3 node cluster running on the SAN so it’s unlikely I’m going to feel much of a performance gain from 10gb and the rest, plus the existing iSCSI controllers have some nice failover/redundancy features.  Good to think about though.. thanks!

      • Miles Erickson

        Of course, if you were a VMware shop, you wouldn’t give up; you’d just use Nexenta instead of Windows Storage Server.

        • Mark Lawrence

          Precisely, I was saying it would just work for Hyper V because of the stated need for a VMware certified SAN, but I missed that you wrote ‘SAN software’.   

  • Mark Lawrence

    This is probably why Microsoft has made windows server 8 capable of live migration between non clustered servers, and without using a SAN. Capability such as this rather undermines EMCs dearly held belief that virtualization effectively belongs to the storage layer.

  • Matt Brown

    Along the virtual cluster front, TIBCO Silver formerly Data Synapse’s DataGrid, is a very cool idea… where you virtualize hardware for virtual machines… processing is split across many pieces of hardware…  I was always mildly obsessed with the idea, but have never deeply looked into (aka implemented) it.  But I suspect it makes sense if you have nodes very high bandwidth interconnections (like a 10Gbps+ switch).  But maybe I just want to justify the purchase of a Cray CX-1.

    I’d be curious to see an analysis of the overhead of all these virtualization/abstraction layers adds to the hardware.  Meaning, does it make more sense to run ESX, hosting 25 VMs, on the abstraction layer provided by TIBCO’s Silver, across three fully stocked IBM BladeCenters with the backend of a three “fibre shelfed-up” NetApp? Or does it just make sense to just have ESX control three hosts contained within the three blades with “on-backplaned” disks?

    I like to think that you could leverage the computing and storage access abilities of racks and racks of ClearCube blades efficiently with TIBCO’s Silver; but who knows?!  I think the truth is, it’s always a “latency-cost based analysis” of introducing the least amount of latency into your computed tasks, and this is probably best served by using as close to physical machines as possible.  If you have a large postgresql query taking place against one 500GB SATA disk, versus that of a RAID 10 with 8 disks, obviously the RAID 10 will return the cursor to postgresql with less latency.  When does leveraging a count of disks on a SAN, or a count of distributed processing introduce too much latency?

  • Anonymous

    If you want to get a little more flexible than DRBD,  you might look at OCFS2.

  • The specialised storage problem with most virtualisation schemes is exactly why Red Hat recently acquired the company behind the open source Gluster product (

    (Yes, I do work for Red Hat, but Gluster is an open source product that anyone is free to use today)

  • Gaurav Sinha

    For your networking needs, probably you can have a look at Vyatta’s virtualized routing platforms as well.

  • So what are the limitations of virtual clusters for which we should not use it?

  •  Thanks for the valuable information and insights you have so provided here. I learned  so much from reading it. I need to read more on this topic…I admiring time and effort you put in your blog.  

  • Rory Alsop

    Thought it would be worth having a link to the Security Stack Exchange blog post on security in virtualised environments: