When I started to build the the new data center I went with the Active Directory name stackoverflow.com. One advantage with this split DNS setup is that you can have internal IPs for certain entries. I knew I would have to duplicate domain entries but I did not mind that and I liked the aesthetic so I made that call.

In most cases I don’t think it is really that bad but we happen to run a popular web site that uses that same name. If the developers have to make URL based calls to the site from the site than it would have to be worked around. Other downsides include that the duplicate entries could be wrong. The vote on Server Fault also seems to be against it. So on reflection, this was simply the wrong call.

So yesterday I went about changing the AD domain name to a sub domain. I also was able to keep STACKOVERFLOW as the short name for the aesthetic. This has been a pain and I think there are still probably some things to fix but in the long run it will pay off.

The moral of the story is that when you make a bad decision then suck it up and fix it. I think this is especially true at the core of a system. If something is wrong there than you have to work around that, and then you have to work around your workaround. What you end up with is:

workaround = "workaround"
while workaround:
    workaround += " the " + workaround

So, if you can, just fix your bad decisions.

I have spent the past couple of months getting our new data center in New York ready to start moving our production applications to. As I have built this out there have been two areas of administration which I have paid particular attention to:

• Scalability of Management
• Fault Tolerance

Scalability of management means to me that I am ahead of the curve when it comes to my method of administration relative to the size of the infrastructure. So if there are 20 servers I want to administer them as if there were 100. The infrastructure right now is:

• 10 Windows IIS Web Servers
• 2 Windows MS SQL Database Servers
• 2 Linux Routers
• 2 Linux Load Balancers with HAProxy
• 1 Linux Backup Server running Bacula
• 1 Linux Management Server for Nagios and Logs
• 2 VMWare ESXi Servers
• 5 Dell Power Connect Switches

It has been a lot of work getting all these operating systems installed, servers wired up (see the image below of one of two cabinets), and everything configured.

However, I have made it into a lot more work that it might have been. Why? It wasn’t because I labeled each end of each cable as much as it is that I have aimed to build out everything in a way that the management of it will scale. More often than not it would have been faster to go and do everything manually on each server. However I have considered that bad for a few main reasons:

If I gain the experience and knowledge of how to manage these servers as if there were at least 3 times as many I won’t have to deal with certain management issues when the pressure is on. I am sure that sort of growth will present all sort of other problems that I will be scrambling to solve. The analogy Jeff Atwood has used for the community problems we have faced as we have grown is big city vs. small city planning. I want to be as ready as possible to face the big city problems. I also believe in the long run this philosophy saves time. Lastly, manually administering each server as a single entity is just lame and pedestrian.

Some of the tools I have used that I believe will aid in scaling my management are as follows:

A Windows Deployment Server has allowed me to set up fully unattended installs of Windows for my servers. I was able to have 10 windows installs with all of them joined to the windows domain in just one morning. Of course it took me about a day to get the unattended configuration working after testing installs on VMs but in the long run I think it is worth it. Deploying servers as we grow will be that much easier.

Group Policies for Windows has been essential to this aspect. These have allowed me to get much of the needed software installed either by using the msi installation feature or by setting up boot up scripts with them. This has included things such as enabling remote desktop, installing software packages such as NSClient++ and Mercurial, configuring firewall rules and deploying roles such as SNMP, IIS and .NET. The software installation aspect really can’t compare to something like apt for Debian but nonetheless it is essential to take advantage of what I can.

Kludgy power shell scripts have been useful to for copying configuration files to all the servers and allowing me to restart the service. Although I have done a lot of bash shell scripting, power shell is new to me. However I was able to copy the NSClient ini file and restart the service using just a little snippet I picked up in not too long:

$webservers = @("ny-web01", "ny-web02", "ny-web03", "ny-web04", "ny-web05", "ny-web06", "ny-web07", "ny-web08", "ny-web09")


foreach ($server in $webservers) {
    Write-Host $server
    cp 'C:Program FilesNSClient++NSC.ini' "\$serverc$Program FilesNSCLient++NSC.ini"
    $svc = Get-WmiObject Win32_Service -ComputerName $server -Filter "name='nsclientpp'"
    if ($svc.started -eq $true) {
        $svc.StopService()
    }
    $svc.StartService()
}

This as well had a higher up front time cost than just manually copying the file, but in the long run it is better administration, it scales, and saves time. I have already had to make some changes to that configuration and pushing the changes now is fast and simple.

I have deployed a Windows Update Server (WSUS) to allow me to control the updates that go to certain servers from a centralized location. I opted to have the updates are scheduled via a group policy and controlled with WSUS by using client-side targeting.

Nagios has also been essential in this philosophy in that it uses an inheritance and template system. Because of this once all the services and groups are configured all I have to do is add the host the group to getting everything monitored. To further extended this I have used n2rrd for graphing which plugs directly into Nagios. This way adding checks and graphing to a new server deployment is all done in a single step. Also if I want to add a new check to all the servers once I get it setup I just have it run for the entire group which is as simple as adding it the host group. Gathering the data in the graph means storing the results over time which is essential for trending so that new hardware can be purchased ahead of time.

All these tools are the tools I think that are needed to build a city. Some tools are good for a server or two but they just don’t scale.

There is still more to do. For the Linux boxes I have been using ssh and loop scripts similar to the power shell script above and really something like Puppet is more appropriate. Also not every piece of software had a simple way to deploy via a group policy. Sadly enough, .NET 4 doesn’t come as an MSI and isn’t available in the update services yet (but is coming “soon”, as in 2-12 months). Also a centralized logging solution is on my to do list as checking the logs on each server is quite tedious.

Fault Tolerance: Fault tolerance is basically the idea that if one component fails it doesn’t bring everything down. I am of the belief the network is a good area to start with fault tolerance and this is where I have put much of my efforts.

Network Fault Tolerance: In a previous Server Fault Blog post entitled “HSRP is not for WANs” I explained why I didn’t think HSRP was valid for the WAN and that a routing protocol should be used. I managed to convince our collocation provider Peer 1 of this and we now have a private BGP peering with them. On our 2 Linux routers I use the Quagga routing suite for BGP. ucarp is used to provide a redundant gateway for the LAN. For stateful firewall rules in IPTables I use conntrackd to make sure that any asymmetric routing patterns don’t break because of mismatched state tables.

Each server is connected to two different switches and has bonding in an active fail over mode.

With all of this going on the key has been to test the fail over by bring down the routers and interfaces on the routers. Without the tests you are only hoping you have fault tolerance.

Power Fault Tolerance in Thirds: Power can be tricky for fail over. Many people think you just put two power supplies in each server connected to independent feeds and you are good to go. However, I had learned from others experiences that at least with the older Dell PowerEdge servers you didn’t know which power supply would be active. If you don’t properly watch your power when one of the feeds fails you can overload the other and have no power. So the alternative I have taken is to divide the servers into groups of 3 (when there are more than two). One server will have dual power supplies, the second will be connected to the A feed and the third server to the B feed. This way if power fails we will be operating at 66% instead of 50% capacity. In general when there are only 2 servers the second is a warm backup so I just make sure they are on different feeds. If there is only one server that I have dual power supplies in it).

Fault Prevention: I have also made sure to set up a staging server in the production environment. The best way to handle faults is to stop them from happening and I think having a test server should cut down on them quite a bit.

I think scalability of management and fault tolerance are just two aspects that make the difference between administering servers and being a professional system administrator. These are the sort of topics that make system administration a profession and it is one reason why Server Fault can be such a great resource for system administrators.

Server Fault now has a place for real time chat over at chat.serverfault.com. We have had a room going for a few weeks now over at meta.chat.stackoverflow.com but it is nice for us to have our own place now where we can create specific chat rooms now.

A place for live chat is a great thing to have for system administrators. A lot of system administrators seem to be out working on their own so this is a nice place to socialize. It also can be a place to talk about those questions that don’t really have straightforward answers. For example:

If you haven’t seen the chat system that Stack Overflow’s employees Benjamin Dumke and Marc Gravell have developed you are missing out. I bet a lot of system administrators, especially grumpy ones with a *nix slant like me, might have a first thought similar to what I had:

Why do we need some Web 2.0 garbage to replace our wonderful text based IRC?

It turns out there a lot of reasons besides all the damn netsplits. This chat retains all the history so when you join a room you can see what was being talked about, you can also search the transcript. There is a lot more as well — I recommend you go check it out.

The very first post on this blog, Experiments in Rate Limiting, explained how our load balancer HAProxy did not support rate limiting. So in order to support rate limiting we placed Nginx in front of HAProxy. This has worked but it wasn’t ideal for a few reasons:

• Having a reverse proxy in front of another one is just a little bit too Rube Goldberg.

• We found that Nginx doesn’t allow rates to be calculated at rates greater than hits per second. So you can write rates as per minute but that is just another way of writing per second rates.
• We wanted the ability to track the amount of data transferred over period of time as well as hits.

So in order to continue the company’s goal which is to make the Internet a better place to get expert answers rate limiting to your questions for your websites we contracted HAProxy’s author Willy Tarreau to add this feature to HAProxy. This solution made sense to us because:

• HAProxy has worked great for us and other companies like reddit.
• Willy is awesome and it so happens that Willy is a Server Fault user.
• Since HAProxy is free and open source our rate limiting is your rate limiting. So head on over and get the code if you like.

After working with Willy over the past month by putting on my testing/QA hat we ended up with a version we felt we could try in production. So on Saturday night it went live and survived the high traffic Monday.

How to Set it up: If you have multiple sites under HAProxy a good way we have found to configure it is to have a frontend/backend configuration. The frontend is what HAProxy listens on and it sends connections to the backend(s) which can be different sites and use different pools of web servers.

This is and a sample of the rate limiting features are best explained with a sample configuration so here is a working configuration that can be used to experiment with:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    stats socket /var/run/haproxy.stat mode 600 level operator
    maxconn 4096
    user haproxy
    group haproxy
    daemon


defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout      5000
    clitimeout      50000
    srvtimeout      50000

frontend http
    bind *:2550

stick-table type ip size 200k expire 10m store gpc0

# check the source before tracking counters, that will allow it to
# expire the entry even if there is still activity.
acl whitelist src 192.168.1.154
acl source_is_abuser src_get_gpc0(http) gt 0
use_backend ease-up-y0 if source_is_abuser
tcp-request connection track-sc1 src if ! source_is_abuser

acl is_test1 hdr_sub(host) -i test1.com
acl is_test2 hdr_sub(host) -i test2.com

use_backend test1  if is_test1
use_backend test2  if is_test2


backend test1
    stick-table type ip   size 200k   expire 30s   store conn_rate(100s),bytes_out_rate(60s)
    acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 3
acl data_rate_abuse  sc2_bytes_out_rate  gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser   sc1_inc_gpc0 gt 0
tcp-request content  reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content  reject if data_rate_abuse mark_as_abuser

server local_apache localhost:80


backend test2
    stick-table type ip size 200k expire 1m store conn_rate(100s),bytes_out_rate(60s)
    acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 5
acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser  sc1_inc_gpc0 gt 0
tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content reject if data_rate_abuse mark_as_abuser
server local_apache localhost:80


backend ease-up-y0
    mode http
    errorfile 503 /etc/haproxy/errors/503rate.http

So the backends both use the same single web server in our test environment which happens to run on the same machine: “server local_apache localhost:80”. test1.com and test2.com are directed to the appropriate backend with “acl is_test1 hdr_sub(host) -i test1.com” which is an access list that returns true if the host header in the HTTP request is test1.com. We then use “use_backend test1 if is_test1” to redirect it to the backend. This is how the basic set up works in a very short summary because I want to focus on the actual rate limiting configuration.

If people violate the rate limiting in this example they are redirected to the backend ease-up-y0 which gives them 503 error page that can be customized. Different backends have different rates in our example but the blocking happens on the front end. Looking first at the backend for test1.com there are several pieces to this. First we track average connection rate and bytes rate per source ip with:

stick-table type ip size 200k expire 3m store conn_rate(100s),bytes_out_rate(60s)

This declares a table to store the source IP addresses that is up to 200,000 entries long. Each IP entry is about 50 bytes and the connection rate and bytes out rate are 12 bytes each which are stored with each source IP address. So at 74 Bytes an entry we are looking at a possible 14 MBytes of usage for this table. The expire argument is how long to keep an entry in the table (In this case it just needs to be twice the length of the longest rate argument for a smoothed average). The time arguments for connection rate and bytes out rate are how long to calculate the average over.

The next part enables the tracking and sets the limits:

    tcp-request content  track-sc2 src
    acl conn_rate_abuse  sc2_conn_rate gt 5
    acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

Finally we have the section that actually enforces these rules:

    acl mark_as_abuser  sc1_inc_gpc0 gt 0
    tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
    tcp-request content reject if data_rate_abuse !whitelist mark_as_abuser

This section is a little bit more cryptic but is not too complicated. The gpc0 is a generic counter and sc1_inc_gpc0 is incrementing the counter we use on the frontend — if it is evaluated. So in the tcp-request line if they have violated either the connection rate or the data rate and they are not whitelisted then the front-end gpc0 will be incremented. The mark_as_abuser does not get evaluated if conn_rate_abuse or data_rate_abuse because this uses short circuit evaluation. This incremented front end gpc is then handled by the frontend:

    stick-table type ip size 200k expire 10m store gpc0
    # check the source before tracking counters, that will allow it to
    # expire the entry even if there is still activity.
    acl whitelist src 192.168.1.154
    acl source_is_abuser src_get_gpc0(http) gt 0
    use_backend ease-up-y0 if source_is_abuser
    tcp-request connection track-sc1 src if ! source_is_abuser

In this stick table we store the general counter with the source ip address. If that has been incremented by a backend the user is redirected to the 503 page. Once this has happened they are not longer tracked during this period, as a result they are redirected to 503 until their entry expires which is set to 10 minutes in this example.

A good way to get a feel for this is to use the socket connection feature that HAProxy has. Here is an example session (Here the client happens to be on localhost so the client entry is 127.0.0.1):

sudo socat readline /var/run/haproxy.stat
prompt
> set timeout cli 1d
> show table


table: http, type: 0, size:2048000, used:1

table: test1, type: 0, size:2048000, used:1

table: test2, type: 0, size:2048000, used:0

> show table http

table: http, type: 0, size:2048000, used:1

0x8370aac: key=127.0.0.1 use=0 exp=592347 gpc0=0

> show table test1

table: test1, type: 0, size:2048000, used:1

0x8370a68: key=127.0.0.1 use=0 exp=19020 conn_rate(100000)=2 bytes_out_rate(60000)=986

> help
Unknown command. Please enter one of the following commands only :
  clear counters : clear max statistics counters (add 'all' for all counters)
  clear table    : remove an entry from a table
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show info      : report information about the running process
  show stat      : report counters for each proxy and server
  show errors    : report last request and response errors for each proxy
  show sess [id] : report the list of current sessions or dump this session
  show table [id]: report table usage stats or dump this table's contents
  get weight     : report a server's current weight
  set weight     : change a server's weight
  set timeout    : change a timeout setting
  disable server : set a server in maintenance mode
  enable server  : re-enable a server that was previously in maintenance mode

The documentation for this new feature set is pretty thorough and will explain all of the above in more detail but I wanted to give a starting point to start experimenting with the rate limiting. I think this is an excellent new feature that introduces some web application firewall features into HAProxy that fit nicely. If you are using HAProxy I recommend giving this new feature a try and if you are not using HAProxy and have a use for a load balancer you should check it out.

My teachers in grade school hammered at least one lesson into me:

“If it is in a book, it must be true”

Since I was such a good student I remember this little aphorism exactly as it was stated even to this day. This is great news for Server Fault as it was mentioned in the 4th Edition of the Unix and Linux System Administration Handbook as one of the most significant resources of UNIX and Linux information on the web. I already felt this was the case myself, but it is good to know that it must be true.

Thank you Zordache for pointing this out in the Server Fault chat room.

A question that comes up again and again in web development companies is:

“Should the developers have access to the production environment, and if they do, to what extent?”

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers — so please don’t take it this way. First I want to cover a few common arguments of developers that dislike or hate this idea:

“We can’t get stuff done, the system administrators get in the way and take forever.” Well if this is actually the case, then they are right. If there are not enough administrators or the administrators are not good then they can become a bottleneck. However, developer access is not the solution because after this you still have crappy or not enough administrators. Sometimes there are other administrative specific concerns that might make things take longer, more on this later, but it shouldn’t take an unreasonable amount of time.

“We have always had access before.” Startup companies seem to rarely start out with administrators. For some reason system administrators are considered a luxury. Although this process might have worked before, as you grow there is probably more administration. Things get more complicated and this is probably why they went out and hired an administrator. So in this case, “this is what we have always done” isn’t really good enough argument.

“We need access to troubleshoot.” Maybe, maybe not. It is possible the administrators can just give you the information you need. If this particular area becomes a bottleneck, limited access might be in order.

Those are a few possible arguments against restricted access for developers, but lets move on to what I really want to talk about — why it is a good idea.

The Process Restricted Access Creates: If the developers can not access production one big implication is that they can’t install their own code. This means that the administrators must install the code. Two things then need to happen: 1) The developers and system administrators must communicate — with each other! The administrators learn how to install the software which I hope I don’t have to explain is probably a good thing. 2) The developers have to create an install or release process that is easy and effective. This is also a good idea. Being able to rebuild the environment is an essential part of disaster recovery. So what can’t happen with restricted access is that the installation of the code is some complex process that only lives in heads of a few developers. Also, the developers don’t have to spend time deploying and installing code when they could be writing new code. It might take them longer at first, but asymptotically this is will be faster (That is right, I used a fancy developer word).

Administrators will also probably learn a little bit more about what needs to be backed up through this process. If the administrator doesn’t know the application well they just have to trust that what the developer told them to back up is all that really needs to be backed up.

Developer’s Concerns are Often Not System Administrator’s Concerns: In general developers do not focus on security in the same areas that system administrators do. They usually have different areas of expertise when it comes to web site security. Topics such as cross site scripting and SQL injection are likely areas of security where developers have specific expertise and administrators do not. Account privileges, file permissions, web server configuration are often not what developers have experience in or are very interested in. These are all important areas in production environments are meant to the expertise of system administrators. Also while I am on the topic of security the less people with access the better (Principle of Least Privilege). This also helps when the call comes in at 2am because the system administrator doesn’t have to wonder if one of the 15 developers with access were on the system doing … something.

Change Control: I don’t think there is a decent developer out there that isn’t serious about change control. When it comes to their code. However, I haven’t seen to many developers that are serious about logging every single change they make to server as a whole (I have seen some configuration files under revision control however).

If this isn’t done it means that the production environment will not be able to be rebuilt properly. It also means that if there have been changes that might have caused a problem those changes might not be know to the person trying to solve the problem. The parallel to this would be if an administrator just went into the production code and changed some things without telling anyone or checking it in. Ya, the developers would freak.

The Person Who Owns it Should Have Control: One of Joel’s Spolsky’s beliefs when it comes to management is:

“Everybody owns some area. When they own it, they own it. If a manager, or anybody else, wants to provide input into how that area is managed, they have to convince the owner. The owner has final say.”

System administrators are generally considered to own the production environment. The administrators are the ones who keep track of uptime, the ones who get the phone calls at 2am, basically, they are the ones closest to the problem. When developers have direct access to production from what I have seen this control always gets undermined.

The System Administrators Responsibilities: In order for this to work, administrators have duties that must be fulfilled. 1) Invite the developers to request what they need from you and be pleasant about giving it to them. 2) Make sure the developers have a good development environment in which they have free rein. 3) Be reasonable and practical. Every company is different, for some companies maybe developers should just have no access because of the nature of the business (i.e. finance). However, if you are not a financial company, a work flow where developers have unprivileged access is likely the best solution. There might also be some developers that double as system administrators so every company has a different situation.

As I stated in the beginning my belief in the process doesn’t have to do with having great or not-so-great developers — many developers function great as system administrators (For example see this post on sending email without it being tagged as spam). Rather this is about a process that lets both people focus on their expertise as a company grows. Things may move a little bit slower. However, the trade in should be that you get a more reliable and secure production environment.

Server Fault now has it’s own meta site at meta.serverfault.com. The idea of the meta site is that it is a place for people who participate on Server Fault to talk about the functioning of the site and the community itself. In the past if you wanted to talk about Server Fault you could do it on Stack Overflow’s meta but in general our system administration specific concerns were quickly buried by those of the larger original Stack Overflow site for programmers. If you are curious about the details of these per-site metas you can read more on the Stack Overflow blog.

I am excited about this new site because I believe as it gains traction it is going to help the community define itself. Server Fault has grown over the past year and has grown to a point where it is ready for the community to take ownership of itself. There has already been a question about how Server Fault appeals to professionals vs. administrators who are just starting out. This new place provides an opportunity for the dedicated members of the community to have conversations they feel address our specific concerns in creating an even better site for professional system administrators to get answers to their questions, become better system administrators, and form a great system administration community.

HSRP or Hot Standby Router Protocol is a protocol designed to provide a virtual gateway for the purposes of fail over. The basic idea is that you have the gateway IP address shared between two or more devices and only one of those devices holds it at a time. If one of those devices fails another device will take hold of this IP address which is commonly referred to as the virtual IP (VIP). Other implementations of this are CARP and VRRP but for this discussion I will just use HSRP.

This solution is great to provide a redundant gateway for hosts on the LAN side of your network which is what it was designed for. RFC 2281 defines HSRP and is pretty clear about what it was designed for:

“Using HSRP, a set of routers work in concert to present the illusion of a single virtual router to the hosts on the LAN.”

“In particular, the protocol protects against the failure of the first hop router when the source host cannot learn the IP address of the first hop router dynamically. … HSRP is not intended as a replacement for existing dynamic router discovery mechanisms and those protocols should be used instead whenever possible.”

Despite this, I have had a couple of occasions where providers have tried to convince me that HSRP will provide the same level of redundancy as a dynamic routing protocol would for routers — in other words beyond the first hop. This setup turns out to be problematic.

The HSRP Setup and the Unemployed Network Administrator:

The setup for full network redundancy that the provider gives to the client’s network administrator has three different instances of HSRP. The first is a virtual gateway for the client from the provider (192.168.0.1). In addition to this the provider has the client run HSRP on the client’s WAN side so the provider can route 172.16.0.0/24 via the VIP 192.168.0.5. The third instance is for the client LAN (172.16.0.1), which is how HSRP is actually supposed to be used.

At first glance this looks fine. The network administrator goes through the diagram and asks himself for every router “What happens if this router fails?”. If Provider 1 fails then Provider 2 picks up the 192.168.0.1 VIP and the layer 2 switches make sure that the packets go to the right ports. The opposite is true if Provider 1 fails. The same basic scenario happens if one of the client routers fails as well. So the network administrator thinks, “Great, it works, I just need to explain this to my boss and then I can go home….” He goes to explain this to his annoying pointy haired boss and after the boss asks “What about the blue boxes?”. The network administrators says, “Oh you must mean the switches”, however he thinks to himself “You are not smart enough to even know what they are called — why do I have to explain this?”

So he hypothetically takes out Switch 1 and explains what happens as fast as he can in hopes to confuse his boss enough that he just gets to home. “In this scenario Provider 2 picks up the VIP of 192.168.0.1 and Client 2 picks up the VIP of 192.168.0.5. The result is that there is still a path to the web server.” After this Mr. Pointy Hair says, “Okay that looks right to me but I do want you to go test it.” The network administrator grumbles and agrees to test it even though he knows it is going to work just fine because not only does he know it will work but the provider also agrees and they do this for a living.

So the network administrator sets it all up and pulls the plug on the switch only to discover that it just doesn’t work — why?! The reason is that although Client 2 changed the WAN VIP, the LAN VIP still remains on Client 1. Therefore replies go to Client 1 and then have no way to reach the gateway on Provider 2. The network administrator for the client calls the provider and starts complaining that he is going to look like an incompetent to his boss now. The provider says they will look into it and then calls back the next day and says, “Don’t worry, you can have your LAN VIP track the WAN VIP so they stay synced up — If the WAN goes down the LAN will follow”. The network administrator does this in test and shuts down Switch 1 and to his great relief everything works. They deploy this in production and live happily ever after…

Until one day switch 3 fails. When this happens the mirror problem of switch 1 failing happens. The web server can send packets out via Client 2 but replies end up stuck on Client 1 because although the LAN tracks the WAN, the WAN doesn’t track the LAN. This problem is found by the network consultant that Mr. Pointy Hair hired to find out what the hell happened. Soon after this Mr. Network Administrator gets canned.

So what is the Solution? You might say the solution is to have both interfaces on the client routers track each other. Personally I never tried this and it might even work. However at this point if you get this to work you really missed the moral of the story. The purpose of HSRP is clearly documented and it tells you to use a dynamic routing protocol such as BGP, OPSF, ISIS, or EIGRP. The people who created this are probably either smarter than you are or spent more time thinking about it. You don’t want to hack it for a WAN by adding extra layer 2 devices and various configuration tricks. Do what the smart people say, if your provider disagrees don’t stop until they give you a proper solution. They can do it no matter what they say, you just have to convince them.

For our datacenter in Corvallis we purchased a new Sans Digital EliteNAS EN104L+XR to replace our slower QNAP NAS.

In this post I want to try to figure out what the theoretical RAID performance of this NAS should be and then compare that to the performance we actually see.

I Present the EliteNAS:
The EliteNAS, which I would have called The l33t NAS, is a Network Attached Storage appliance built on top of GNU/Linux thats boasts the following:

• 1U enclosure
• Linux OS with custom web UI
• Intel Xeon 3040 1.84GHz Dual Core CPU
• dual gigabit ethernet
• hot-swappable redundant power supplies
• four SATA ports (software RAID)

We have installed 4×1.5TB drives into this configured in a RAID 10 configuration in hopes to see a significant gain in performance over our previous RAID 6 configuration.

This device also supports gigabit bonding. When they said that it is built on top of Linux, they mean it. All of the Linux bonding options are available through the GUI, Linux Software RAID is used, and you get access to the CLI via ssh if you are so inclined, which I must say I am:

[root@sonas1 ~]# /sbin/mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Thu Jul  8 03:25:55 2010
     Raid Level : raid10
     Array Size : 2930272128 (2794.53 GiB 3000.60 GB)
    Device Size : 1465136064 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
...
         Layout : near=2, far=1
     Chunk Size : 64K
...
    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

The file system used is XFS with the noatime and nodiratime mount options so there is not a write to the file every time it is read. It also comes with all the standard unix tools such as tcpdump and iostat.

As for the drives themselves we have 4 Western Digital Caviar Greens (WD15EARS). These are 3Gb/s SATA drives with a 64MB cache. Tom’s hardware has a nice review of these drives which I am going to rely on for individual drive performance. The highlights from the tests are (Ballpark Avg from Multiple Benchmark Tools):

Avg. IOPS: ~120
Avg. Sequential Throughput (Same for reads and writes): ~94 MBytes/s

What is RAID 10?
With Raid 10 some people might have discovered that there is RAID 1+0 and RAID 0+1 and you can find the difference between these talked about in this serverfault question. In this case we are using RAID 1+0:

So we have two sets of mirrors with the data striped over them. With this configuration we can lose up to two drives as long as both of those drives are not in same mirror.

What should our Theoretical Speeds of Raid 10 Be?
So I am lucky enough that Tom’s Hardware decided to go ahead and take care of the individual disk performance benchmarking as I already mentioned. So how do we figure out what happens in terms of performance when we combine them in RAID 10?

With RAID 10 the reads go over the stripe (RAID 0). So looking at our RAID 10 diagram we are reading from two RAID 1 mirrors. In this case we expect to see the sequential read performance of about two drives in RAID 0 according to the mdadm maintainer Niel Brown. If this was in the far layout configuration of RAID 10 we would get read performance on par with a 4 drive RAID 0 array but would incur a larger write penalty. For writes since we have a mirror every logical write causes two physical writes. However we have two sets of mirrors so I believe that we can expect the write performance of about a single disk. However as we will see from my benchmarks that turns out not to be the case.

Since the purpose of this device is for backups we are essentially interested in sequential disk activity because we will be streaming large contiguous files to and from the array.

Actual Read Performance:

Device:  rrqm/s    wrqm/s  r/s       w/s   rsec/s     wsec/s  rkB/s      wkB/s  avgrq-sz  avgqu-sz  await  svctm  %util
sda      7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sda1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sda2     7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sdb      9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdb1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdb2     9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdc      5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdc1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdc2     5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdd      10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sdd1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdd2     10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sde      0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sde1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
md1      0.00      0.00    33109.33  0.00  264874.67  0.00    132437.33  0.00   8.00      0.00      0.00   0.00   0.00

For sequential reading I get about 130 MBytes/s. The single drive benchmarks from Tom’s hardware were about 90 MBytes/s. So for sequential reads there is a (130-90)/90 or 44% increase over one disk. When reading over the network through SMB read performance is about 105 MB/s when copying to the disk of a remote server. So for us that means that either SMB adds an overhead of about 25 MB/s, we are hitting a network bottleneck, or that there is a write bottleneck on the destination server. This NAS has the ability to bond the interfaces so down the road I plan to implement this as see if there is any performance improvement.

Actual Write Performance:
Measuring writes with iostat doesn’t seem to work as well because I believe the writes are cached and serialized by the md device. Therefore the numbers jump around a lot. However I believe dd with a large block size and a timed write provides a reasonable benchmark:

[root@sonas1 test]# time dd of=testfile if=/dev/zero bs=1M count=100000008008+0 records in
8007+0 records out
real    2m48.763s
....
[root@sonas1 test]# echo $((8007/168))
47

So for writes we see 47MBytes/s. I tested this again with 64K block size and did see an improvement of 63MBytes/s. According to Tom’s hardware a single drive should put through about 90 MBytes/s for writes. So in this there is a (90-63)/90 or 30% decrease in sequential write performance for the 4 disk RAID 10 array in comparison to a single disk.

Oh Noes! My Theory didn’t hold up!
So there are a few possibilities here:

• My benchmarking of sequential performance is flawed.
• There is a bottleneck I am missing.
• Something is wrong with the configuration.
• My theory was wrong and I need to change it.

The first thing I did was research my benchmarking methodology of using dd to test sequential performance. I found that some people don’t recommend it. However, I used different block sizes which is one of the main concerns. Also dd writes sequentially and in our case that is what this device is used for. I was also careful to always flush my buffers before each test. Therefore I am happy with my benchmark to provide a reasonable ballpark estimate of performance for our intended workload.

For a bottleneck the two things I considered were CPU and the bus speed. According to the infallible Wikipedia 3 Gb/s SATA goes up to 286.10 MByte/s so I don’t believe that is a bottleneck. I watched CPU load during these tests and there was no significant load on either core.

For the possibility of something being wrong with the configuration I would love to hear if anyone has any recommendations. However, this is an appliance at heart so we did all the building through the Web GUI so I am not sure tweaking it is the best idea. One possibility is that there is a partition alignment issue with the these disks which have 4096 byte sectors. I am going to investigate this on my own and have also posted a question on Server Fault in hopes that someone in the community might already have the expert knowledge needed to diagnose if this is an issue.

That leaves the possibility that my theoretical expectations were wrong which seems to be the most likely. For the writes the extra 44% seems to fall short of what I would expect. I would have guessed that there would be about a 80% increase over a single disk. The idea that the write penalty due to mirroring would be cancelled out by the striping also turned out not to be true. The 30% hit also seems like a bigger hit than I would have expected.

The only real numbers of RAID 10 performance relative to a single disk that I could find were in the zdnet article Comprehensive Raid Performance Report. In the graphs comparing RAID 10 of 4 drives to the performance of a single drive I see a slight increase of write performance and a 100% increase in writes. However, it is not clear to me if this is random activity or sequential. I would expect that RAID might be a bigger advantage in random IO because of seek time.

Conclusion:
In the end this turns out to be a lot faster than our previous NAS and backups that took 25 minutes now take 5 minutes so we are happy with the upgrade. So I say farewell to our old NAS which had been given our love:

When it comes to predicting RAID performance based off of a single drive I would love to see more data for both sequential and random patterns:

• How do different drives and RAID cards perform?
• What is the formula of increase for RAID 10 and RAID 6 when you go from 4 drives to 6, 6 to 8, etc?
• How do different RAID levels actually compare in terms of single drive performance?

I think more comprehensive data could lead to betters formulas for estimating RAID performance. If these become available publicly on the net it could really help out system administration and turn RAID choice into a little bit less of a guessing game when it comes to performance.

In this post I am going to explore how extended iostat statistics can be useful to a system administrator beyond a binary “Disk is bottleneck / Disk is not bottleneck.” Before we can get to any of that however, we must make sure we have a basic background knowledge of the Disk IO Subsystem.

Linux Disk IO Subsystem:

I am not a kernel hacker, so this overview might be flawed in parts but hopefully it is accurate enough to give the background needed for analyzing the output of iostat.

Layer	Unit	Typical Unit Size
User Space System Calls	read() , write()
Virtual File System Switch (VFS)	Block	4096 Bytes
Disk Caches	Page
Filesystem (For example ext3)	Blocks	4096 Bytes (Can be set at FS creation)
Generic Block Layer	Page Frames / Block IO Operations (bio)
I/O Scheduler Layer	bios per block device (Which this layer may combine)
Block Device Driver	Segment	512 Bytes
Hard Disk	Sector	512 Bytes

There are two basic system calls, read() and write(), that a user process can make to read data from a file system. In the kernel these are handled by the Linux Virtual Filesystem Switch (VFS). VFS is an abstraction to all file systems so they look the same to the user space and it also handles the interface between the file system and the block device layer. The caching layer provides caching of disk reads and writes in terms of memory pages. The generic block layer breaks down IO operations that might involve many different non-contiguous blocks into multiple IO operations. The I/O scheduling layer takes these IO operations and schedules them based on order on disk, priority, and/or direction. Lastly, the device driver handles interfacing with the hardware for the actual operations in terms of disk sectors which are usually 512 bytes.

A Little Bit on Page Caching:

The page cache caches pages of data that do or will reside on disk. Therefore before it writes data to disk it puts it in memory, and before it reads data from disk it checks to see if it is in memory already (With the exception of Direct IO). Writing pages out to disk actually gets deferred. This is done to increase performance so writes can be grouped together more efficiently. When a page of disk data gets changed and needs to be written out to disk it is called “dirty”. Since it is dangerous to keep pages in memory for too long in case of a system shutdown the kernel’s pdflush threads scan for dirty pages and then flushes them out to disk. Linux will actually try to use as much memory as it can for caching files which is why the top command usually shows so much used memory. When you want to see how much memory is free for processes you can run the free command and look at the ‘-/+ buffers/cache’.

iostat output:

So with this background lets look at some of the output of iostat and tie it together with our background knowledge. Iostat can break down the statistics at both the partition level and then device level, however in this post I am going to focus on the device level.

The Overview Statistics: “Is it Saturated or Not?”

From iostat there are two summary statistics which are Input/Output CPU wait time (iowait) and device utilization which are both expressed in terms of percentages.

iowait is from the CPU’s perspective and it is the percentage of time that the CPU spent waiting for a IO device to be ready. Another way to look at iowait is the amount of time that the CPU could have been doing something but couldn’t because all the processes were waiting on the disk or the network devices.

Device utilization is covered throughly by Alex Gorbahev in Basic I/O Monitoring on Linux. He summarizes it as “The percentage of time the device spent servicing requests as opposed to being idle.”

iostat and caching:

It is import to note that iostat shows requests to the device (or partition) and not read and write requests from user space. So in the table above iostat is reading below the disk cache layer. Therefore, iostat says noting about your cache hit ratio for block devices. So it is possible that disk IO problems might be able to be resolved by memory upgrades. From my research there is no way to pull out a cache hit/miss ratio out of Linux easily when it comes to block devices which is a bit disappointing.  One suggestion from serverfault is to install a kernel with debuging symbols and use SystemTap to trace the VFS events and tie them together with the block layer events. I intend to explore this but I would prefer to see a way to get this data from /proc or /sys.

iostat Output for Random and Sequential Reads:

One of the main things to do when examining disk IO is to determine if the disk access patterns are sequential or random. This information can aid in our disk choices. When operations are random the seek time of the disk becomes more important. This is because physically the drive head has to jump around. Seek time is the measurement of the speed at which the heads can do this. For small random reads solid state disks can be a huge advantage.

So in fio I have created two different simple tests to run. The first is sequential reading, and the second is random reading. During these tests I ran iostat -x 3 throughout the test.

Snapshot of Random Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  172.67    0.00  1381.33     0.00     8.00     0.99    5.76   5.76  99.47

Snapshot of Sequential Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              13.00     0.00  367.00    0.00 151893.33     0.00   413.88     2.46    6.71   2.72 100.00

What is more important to me for this is not just what these numbers are but what, in the context of of random vs sequential reading and in context of the IO subsystem, they mean.

The first two columns, rrqm/s and wrqm/s, are read and write requests merged per second. In my above diagram of the Linux Block IO subsystem above I mentioned that that the scheduler can combine operations. This can be done when multiple operations are physically adjacent to each other on the device. So in sequential operation it would make sense to often see a large number of merges. In the snapshot of the random reads, we see no merges. However, the merging layer feels a little bit like “magic” and I don’t believe it is the best indicator of if the patterns are random or sequential.

The next 5 columns are read and write requests to the device (r/s, w/s), followed by the amount of sectors read and written from the device (rsec/s, wsec/s), and then the size of each request (avgrq-sz). In the random test there are 172 reads that result in 1,381 sectors being read in. In the sequential test there are 367 read request to 151,893 sectors being read. So in the random test we get about 8 sectors per request and in the sequential test we get 413 sectors per read. If you look closely, this happens to be the same number as avgrq-sz which does this math for us (Sectors Read / Read Operations). However it is worth noting that this is how it is calculated as the average request size does not differentiate between reads and writes. From these tests a low sector write/read to request ratio or small request sizes seem to indicate a random IO profile. I believe this to be a better indicator than the amount of merges as to whether or not there is random or sequential disk patterns.

The final 4 columns are the average queue length of requests to the device (avgqu-sz), how long requests took to be serviced including their time in the queue (await), how long requests took to be serviced by the device after they left the queue (svctm), and lastly the utilization percentage which I already mentioned in the overview statistics section. In the above example random requests take longer for the disk to service as expected because of the seek time. However, the queue itself ends up being shorter which I am unable to explain. Utilization, in more detail, is the service time in ms * total IO operations / 1000 ms. This gives the percentage of how busy the single disk was during the given time slice. I believe for a given utilization level a higher number of operations is probably indicative of a sequential pattern.

I have run various variations on the above. They include a mixture of reads and writes for both random and sequential data as well as sequential and random writes. For the writes I got similar results as far as the ratios were concerned and queue and services time were higher.

In the end it seems average request size is the key to show if the disk usage patterns are random or not since this is post merging. Taking this into the context of the layers above this might not mirror what an application is doing. This is because a read or write operations coming from user space might operate on a fragmented file in which case the generic block layer will break it up and it appears as random disk activity.

Conclusion:

As far as I am concerned this is only a start in interpreting IO statistics. I think these tests need to be repeated, perhaps with different tools to generate the disk IO, as my interpretations might just be totally off. Also, a pretty big limitation of what I did is that my work was all on a single disk and these numbers might have different results under various RAID configurations. I feel the inability to measure the cache hit ratio of reads on a block device is a significant shortcoming that I would love to see addressed since from a system administrators perspective the solution to certain IO problems might be to throw more memory at the problem.

Lastly, I want to make a point about these sort of low level statistics in general. Everything needs to monitored from the an application perspective as well. These statistics can be misleading and are most useful when they can be correlated with the data that actually matters to users of the applications, for example, response time from the user perspective. These also need to be monitored over time because you want to be able to see changes for capacity planning as well as to give them context to past performance when problems arise.

Further Reading:
http://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/
Understanding The Linux Kernel, Chapter 14
http://www.ufsdump.org/papers/io-tuning.pdf
http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/
http://www.kernel.org/doc/Documentation/iostats.txt