The very first post on this blog, Experiments in Rate Limiting, explained how our load balancer HAProxy did not support rate limiting. So in order to support rate limiting we placed Nginx in front of HAProxy. This has worked but it wasn’t ideal for a few reasons:

• Having a reverse proxy in front of another one is just a little bit too Rube Goldberg.

• We found that Nginx doesn’t allow rates to be calculated at rates greater than hits per second. So you can write rates as per minute but that is just another way of writing per second rates.
• We wanted the ability to track the amount of data transferred over period of time as well as hits.

So in order to continue the company’s goal which is to make the Internet a better place to get expert answers rate limiting to your questions for your websites we contracted HAProxy’s author Willy Tarreau to add this feature to HAProxy. This solution made sense to us because:

• HAProxy has worked great for us and other companies like reddit.
• Willy is awesome and it so happens that Willy is a Server Fault user.
• Since HAProxy is free and open source our rate limiting is your rate limiting. So head on over and get the code if you like.

After working with Willy over the past month by putting on my testing/QA hat we ended up with a version we felt we could try in production. So on Saturday night it went live and survived the high traffic Monday.

How to Set it up: If you have multiple sites under HAProxy a good way we have found to configure it is to have a frontend/backend configuration. The frontend is what HAProxy listens on and it sends connections to the backend(s) which can be different sites and use different pools of web servers.

This is and a sample of the rate limiting features are best explained with a sample configuration so here is a working configuration that can be used to experiment with:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    stats socket /var/run/haproxy.stat mode 600 level operator
    maxconn 4096
    user haproxy
    group haproxy
    daemon


defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout      5000
    clitimeout      50000
    srvtimeout      50000

frontend http
    bind *:2550

stick-table type ip size 200k expire 10m store gpc0

# check the source before tracking counters, that will allow it to
# expire the entry even if there is still activity.
acl whitelist src 192.168.1.154
acl source_is_abuser src_get_gpc0(http) gt 0
use_backend ease-up-y0 if source_is_abuser
tcp-request connection track-sc1 src if ! source_is_abuser

acl is_test1 hdr_sub(host) -i test1.com
acl is_test2 hdr_sub(host) -i test2.com

use_backend test1  if is_test1
use_backend test2  if is_test2


backend test1
    stick-table type ip   size 200k   expire 30s   store conn_rate(100s),bytes_out_rate(60s)
    acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 3
acl data_rate_abuse  sc2_bytes_out_rate  gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser   sc1_inc_gpc0 gt 0
tcp-request content  reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content  reject if data_rate_abuse mark_as_abuser

server local_apache localhost:80


backend test2
    stick-table type ip size 200k expire 1m store conn_rate(100s),bytes_out_rate(60s)
    acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 5
acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser  sc1_inc_gpc0 gt 0
tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content reject if data_rate_abuse mark_as_abuser
server local_apache localhost:80


backend ease-up-y0
    mode http
    errorfile 503 /etc/haproxy/errors/503rate.http

So the backends both use the same single web server in our test environment which happens to run on the same machine: “server local_apache localhost:80”. test1.com and test2.com are directed to the appropriate backend with “acl is_test1 hdr_sub(host) -i test1.com” which is an access list that returns true if the host header in the HTTP request is test1.com. We then use “use_backend test1 if is_test1” to redirect it to the backend. This is how the basic set up works in a very short summary because I want to focus on the actual rate limiting configuration.

If people violate the rate limiting in this example they are redirected to the backend ease-up-y0 which gives them 503 error page that can be customized. Different backends have different rates in our example but the blocking happens on the front end. Looking first at the backend for test1.com there are several pieces to this. First we track average connection rate and bytes rate per source ip with:

stick-table type ip size 200k expire 3m store conn_rate(100s),bytes_out_rate(60s)

This declares a table to store the source IP addresses that is up to 200,000 entries long. Each IP entry is about 50 bytes and the connection rate and bytes out rate are 12 bytes each which are stored with each source IP address. So at 74 Bytes an entry we are looking at a possible 14 MBytes of usage for this table. The expire argument is how long to keep an entry in the table (In this case it just needs to be twice the length of the longest rate argument for a smoothed average). The time arguments for connection rate and bytes out rate are how long to calculate the average over.

The next part enables the tracking and sets the limits:

    tcp-request content  track-sc2 src
    acl conn_rate_abuse  sc2_conn_rate gt 5
    acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

Finally we have the section that actually enforces these rules:

    acl mark_as_abuser  sc1_inc_gpc0 gt 0
    tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
    tcp-request content reject if data_rate_abuse !whitelist mark_as_abuser

This section is a little bit more cryptic but is not too complicated. The gpc0 is a generic counter and sc1_inc_gpc0 is incrementing the counter we use on the frontend — if it is evaluated. So in the tcp-request line if they have violated either the connection rate or the data rate and they are not whitelisted then the front-end gpc0 will be incremented. The mark_as_abuser does not get evaluated if conn_rate_abuse or data_rate_abuse because this uses short circuit evaluation. This incremented front end gpc is then handled by the frontend:

    stick-table type ip size 200k expire 10m store gpc0
    # check the source before tracking counters, that will allow it to
    # expire the entry even if there is still activity.
    acl whitelist src 192.168.1.154
    acl source_is_abuser src_get_gpc0(http) gt 0
    use_backend ease-up-y0 if source_is_abuser
    tcp-request connection track-sc1 src if ! source_is_abuser

In this stick table we store the general counter with the source ip address. If that has been incremented by a backend the user is redirected to the 503 page. Once this has happened they are not longer tracked during this period, as a result they are redirected to 503 until their entry expires which is set to 10 minutes in this example.

A good way to get a feel for this is to use the socket connection feature that HAProxy has. Here is an example session (Here the client happens to be on localhost so the client entry is 127.0.0.1):

sudo socat readline /var/run/haproxy.stat
prompt
> set timeout cli 1d
> show table


table: http, type: 0, size:2048000, used:1

table: test1, type: 0, size:2048000, used:1

table: test2, type: 0, size:2048000, used:0

> show table http

table: http, type: 0, size:2048000, used:1

0x8370aac: key=127.0.0.1 use=0 exp=592347 gpc0=0

> show table test1

table: test1, type: 0, size:2048000, used:1

0x8370a68: key=127.0.0.1 use=0 exp=19020 conn_rate(100000)=2 bytes_out_rate(60000)=986

> help
Unknown command. Please enter one of the following commands only :
  clear counters : clear max statistics counters (add 'all' for all counters)
  clear table    : remove an entry from a table
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show info      : report information about the running process
  show stat      : report counters for each proxy and server
  show errors    : report last request and response errors for each proxy
  show sess [id] : report the list of current sessions or dump this session
  show table [id]: report table usage stats or dump this table's contents
  get weight     : report a server's current weight
  set weight     : change a server's weight
  set timeout    : change a timeout setting
  disable server : set a server in maintenance mode
  enable server  : re-enable a server that was previously in maintenance mode

The documentation for this new feature set is pretty thorough and will explain all of the above in more detail but I wanted to give a starting point to start experimenting with the rate limiting. I think this is an excellent new feature that introduces some web application firewall features into HAProxy that fit nicely. If you are using HAProxy I recommend giving this new feature a try and if you are not using HAProxy and have a use for a load balancer you should check it out.

My teachers in grade school hammered at least one lesson into me:

“If it is in a book, it must be true”

Since I was such a good student I remember this little aphorism exactly as it was stated even to this day. This is great news for Server Fault as it was mentioned in the 4th Edition of the Unix and Linux System Administration Handbook as one of the most significant resources of UNIX and Linux information on the web. I already felt this was the case myself, but it is good to know that it must be true.

Thank you Zordache for pointing this out in the Server Fault chat room.

A question that comes up again and again in web development companies is:

“Should the developers have access to the production environment, and if they do, to what extent?”

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers — so please don’t take it this way. First I want to cover a few common arguments of developers that dislike or hate this idea:

“We can’t get stuff done, the system administrators get in the way and take forever.” Well if this is actually the case, then they are right. If there are not enough administrators or the administrators are not good then they can become a bottleneck. However, developer access is not the solution because after this you still have crappy or not enough administrators. Sometimes there are other administrative specific concerns that might make things take longer, more on this later, but it shouldn’t take an unreasonable amount of time.

“We have always had access before.” Startup companies seem to rarely start out with administrators. For some reason system administrators are considered a luxury. Although this process might have worked before, as you grow there is probably more administration. Things get more complicated and this is probably why they went out and hired an administrator. So in this case, “this is what we have always done” isn’t really good enough argument.

“We need access to troubleshoot.” Maybe, maybe not. It is possible the administrators can just give you the information you need. If this particular area becomes a bottleneck, limited access might be in order.

Those are a few possible arguments against restricted access for developers, but lets move on to what I really want to talk about — why it is a good idea.

The Process Restricted Access Creates: If the developers can not access production one big implication is that they can’t install their own code. This means that the administrators must install the code. Two things then need to happen: 1) The developers and system administrators must communicate — with each other! The administrators learn how to install the software which I hope I don’t have to explain is probably a good thing. 2) The developers have to create an install or release process that is easy and effective. This is also a good idea. Being able to rebuild the environment is an essential part of disaster recovery. So what can’t happen with restricted access is that the installation of the code is some complex process that only lives in heads of a few developers. Also, the developers don’t have to spend time deploying and installing code when they could be writing new code. It might take them longer at first, but asymptotically this is will be faster (That is right, I used a fancy developer word).

Administrators will also probably learn a little bit more about what needs to be backed up through this process. If the administrator doesn’t know the application well they just have to trust that what the developer told them to back up is all that really needs to be backed up.

Developer’s Concerns are Often Not System Administrator’s Concerns: In general developers do not focus on security in the same areas that system administrators do. They usually have different areas of expertise when it comes to web site security. Topics such as cross site scripting and SQL injection are likely areas of security where developers have specific expertise and administrators do not. Account privileges, file permissions, web server configuration are often not what developers have experience in or are very interested in. These are all important areas in production environments are meant to the expertise of system administrators. Also while I am on the topic of security the less people with access the better (Principle of Least Privilege). This also helps when the call comes in at 2am because the system administrator doesn’t have to wonder if one of the 15 developers with access were on the system doing … something.

Change Control: I don’t think there is a decent developer out there that isn’t serious about change control. When it comes to their code. However, I haven’t seen to many developers that are serious about logging every single change they make to server as a whole (I have seen some configuration files under revision control however).

If this isn’t done it means that the production environment will not be able to be rebuilt properly. It also means that if there have been changes that might have caused a problem those changes might not be know to the person trying to solve the problem. The parallel to this would be if an administrator just went into the production code and changed some things without telling anyone or checking it in. Ya, the developers would freak.

The Person Who Owns it Should Have Control: One of Joel’s Spolsky’s beliefs when it comes to management is:

“Everybody owns some area. When they own it, they own it. If a manager, or anybody else, wants to provide input into how that area is managed, they have to convince the owner. The owner has final say.”

System administrators are generally considered to own the production environment. The administrators are the ones who keep track of uptime, the ones who get the phone calls at 2am, basically, they are the ones closest to the problem. When developers have direct access to production from what I have seen this control always gets undermined.

The System Administrators Responsibilities: In order for this to work, administrators have duties that must be fulfilled. 1) Invite the developers to request what they need from you and be pleasant about giving it to them. 2) Make sure the developers have a good development environment in which they have free rein. 3) Be reasonable and practical. Every company is different, for some companies maybe developers should just have no access because of the nature of the business (i.e. finance). However, if you are not a financial company, a work flow where developers have unprivileged access is likely the best solution. There might also be some developers that double as system administrators so every company has a different situation.

As I stated in the beginning my belief in the process doesn’t have to do with having great or not-so-great developers — many developers function great as system administrators (For example see this post on sending email without it being tagged as spam). Rather this is about a process that lets both people focus on their expertise as a company grows. Things may move a little bit slower. However, the trade in should be that you get a more reliable and secure production environment.

Server Fault now has it’s own meta site at meta.serverfault.com. The idea of the meta site is that it is a place for people who participate on Server Fault to talk about the functioning of the site and the community itself. In the past if you wanted to talk about Server Fault you could do it on Stack Overflow’s meta but in general our system administration specific concerns were quickly buried by those of the larger original Stack Overflow site for programmers. If you are curious about the details of these per-site metas you can read more on the Stack Overflow blog.

I am excited about this new site because I believe as it gains traction it is going to help the community define itself. Server Fault has grown over the past year and has grown to a point where it is ready for the community to take ownership of itself. There has already been a question about how Server Fault appeals to professionals vs. administrators who are just starting out. This new place provides an opportunity for the dedicated members of the community to have conversations they feel address our specific concerns in creating an even better site for professional system administrators to get answers to their questions, become better system administrators, and form a great system administration community.

HSRP or Hot Standby Router Protocol is a protocol designed to provide a virtual gateway for the purposes of fail over. The basic idea is that you have the gateway IP address shared between two or more devices and only one of those devices holds it at a time. If one of those devices fails another device will take hold of this IP address which is commonly referred to as the virtual IP (VIP). Other implementations of this are CARP and VRRP but for this discussion I will just use HSRP.

This solution is great to provide a redundant gateway for hosts on the LAN side of your network which is what it was designed for. RFC 2281 defines HSRP and is pretty clear about what it was designed for:

“Using HSRP, a set of routers work in concert to present the illusion of a single virtual router to the hosts on the LAN.”

“In particular, the protocol protects against the failure of the first hop router when the source host cannot learn the IP address of the first hop router dynamically. … HSRP is not intended as a replacement for existing dynamic router discovery mechanisms and those protocols should be used instead whenever possible.”

Despite this, I have had a couple of occasions where providers have tried to convince me that HSRP will provide the same level of redundancy as a dynamic routing protocol would for routers — in other words beyond the first hop. This setup turns out to be problematic.

The HSRP Setup and the Unemployed Network Administrator:

The setup for full network redundancy that the provider gives to the client’s network administrator has three different instances of HSRP. The first is a virtual gateway for the client from the provider (192.168.0.1). In addition to this the provider has the client run HSRP on the client’s WAN side so the provider can route 172.16.0.0/24 via the VIP 192.168.0.5. The third instance is for the client LAN (172.16.0.1), which is how HSRP is actually supposed to be used.

At first glance this looks fine. The network administrator goes through the diagram and asks himself for every router “What happens if this router fails?”. If Provider 1 fails then Provider 2 picks up the 192.168.0.1 VIP and the layer 2 switches make sure that the packets go to the right ports. The opposite is true if Provider 1 fails. The same basic scenario happens if one of the client routers fails as well. So the network administrator thinks, “Great, it works, I just need to explain this to my boss and then I can go home….” He goes to explain this to his annoying pointy haired boss and after the boss asks “What about the blue boxes?”. The network administrators says, “Oh you must mean the switches”, however he thinks to himself “You are not smart enough to even know what they are called — why do I have to explain this?”

So he hypothetically takes out Switch 1 and explains what happens as fast as he can in hopes to confuse his boss enough that he just gets to home. “In this scenario Provider 2 picks up the VIP of 192.168.0.1 and Client 2 picks up the VIP of 192.168.0.5. The result is that there is still a path to the web server.” After this Mr. Pointy Hair says, “Okay that looks right to me but I do want you to go test it.” The network administrator grumbles and agrees to test it even though he knows it is going to work just fine because not only does he know it will work but the provider also agrees and they do this for a living.

So the network administrator sets it all up and pulls the plug on the switch only to discover that it just doesn’t work — why?! The reason is that although Client 2 changed the WAN VIP, the LAN VIP still remains on Client 1. Therefore replies go to Client 1 and then have no way to reach the gateway on Provider 2. The network administrator for the client calls the provider and starts complaining that he is going to look like an incompetent to his boss now. The provider says they will look into it and then calls back the next day and says, “Don’t worry, you can have your LAN VIP track the WAN VIP so they stay synced up — If the WAN goes down the LAN will follow”. The network administrator does this in test and shuts down Switch 1 and to his great relief everything works. They deploy this in production and live happily ever after…

Until one day switch 3 fails. When this happens the mirror problem of switch 1 failing happens. The web server can send packets out via Client 2 but replies end up stuck on Client 1 because although the LAN tracks the WAN, the WAN doesn’t track the LAN. This problem is found by the network consultant that Mr. Pointy Hair hired to find out what the hell happened. Soon after this Mr. Network Administrator gets canned.

So what is the Solution? You might say the solution is to have both interfaces on the client routers track each other. Personally I never tried this and it might even work. However at this point if you get this to work you really missed the moral of the story. The purpose of HSRP is clearly documented and it tells you to use a dynamic routing protocol such as BGP, OPSF, ISIS, or EIGRP. The people who created this are probably either smarter than you are or spent more time thinking about it. You don’t want to hack it for a WAN by adding extra layer 2 devices and various configuration tricks. Do what the smart people say, if your provider disagrees don’t stop until they give you a proper solution. They can do it no matter what they say, you just have to convince them.

For our datacenter in Corvallis we purchased a new Sans Digital EliteNAS EN104L+XR to replace our slower QNAP NAS.

In this post I want to try to figure out what the theoretical RAID performance of this NAS should be and then compare that to the performance we actually see.

I Present the EliteNAS:
The EliteNAS, which I would have called The l33t NAS, is a Network Attached Storage appliance built on top of GNU/Linux thats boasts the following:

• 1U enclosure
• Linux OS with custom web UI
• Intel Xeon 3040 1.84GHz Dual Core CPU
• dual gigabit ethernet
• hot-swappable redundant power supplies
• four SATA ports (software RAID)

We have installed 4×1.5TB drives into this configured in a RAID 10 configuration in hopes to see a significant gain in performance over our previous RAID 6 configuration.

This device also supports gigabit bonding. When they said that it is built on top of Linux, they mean it. All of the Linux bonding options are available through the GUI, Linux Software RAID is used, and you get access to the CLI via ssh if you are so inclined, which I must say I am:

[root@sonas1 ~]# /sbin/mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Thu Jul  8 03:25:55 2010
     Raid Level : raid10
     Array Size : 2930272128 (2794.53 GiB 3000.60 GB)
    Device Size : 1465136064 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
...
         Layout : near=2, far=1
     Chunk Size : 64K
...
    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

The file system used is XFS with the noatime and nodiratime mount options so there is not a write to the file every time it is read. It also comes with all the standard unix tools such as tcpdump and iostat.

As for the drives themselves we have 4 Western Digital Caviar Greens (WD15EARS). These are 3Gb/s SATA drives with a 64MB cache. Tom’s hardware has a nice review of these drives which I am going to rely on for individual drive performance. The highlights from the tests are (Ballpark Avg from Multiple Benchmark Tools):

Avg. IOPS: ~120
Avg. Sequential Throughput (Same for reads and writes): ~94 MBytes/s

What is RAID 10?
With Raid 10 some people might have discovered that there is RAID 1+0 and RAID 0+1 and you can find the difference between these talked about in this serverfault question. In this case we are using RAID 1+0:

So we have two sets of mirrors with the data striped over them. With this configuration we can lose up to two drives as long as both of those drives are not in same mirror.

What should our Theoretical Speeds of Raid 10 Be?
So I am lucky enough that Tom’s Hardware decided to go ahead and take care of the individual disk performance benchmarking as I already mentioned. So how do we figure out what happens in terms of performance when we combine them in RAID 10?

With RAID 10 the reads go over the stripe (RAID 0). So looking at our RAID 10 diagram we are reading from two RAID 1 mirrors. In this case we expect to see the sequential read performance of about two drives in RAID 0 according to the mdadm maintainer Niel Brown. If this was in the far layout configuration of RAID 10 we would get read performance on par with a 4 drive RAID 0 array but would incur a larger write penalty. For writes since we have a mirror every logical write causes two physical writes. However we have two sets of mirrors so I believe that we can expect the write performance of about a single disk. However as we will see from my benchmarks that turns out not to be the case.

Since the purpose of this device is for backups we are essentially interested in sequential disk activity because we will be streaming large contiguous files to and from the array.

Actual Read Performance:

Device:  rrqm/s    wrqm/s  r/s       w/s   rsec/s     wsec/s  rkB/s      wkB/s  avgrq-sz  avgqu-sz  await  svctm  %util
sda      7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sda1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sda2     7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sdb      9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdb1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdb2     9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdc      5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdc1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdc2     5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdd      10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sdd1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdd2     10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sde      0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sde1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
md1      0.00      0.00    33109.33  0.00  264874.67  0.00    132437.33  0.00   8.00      0.00      0.00   0.00   0.00

For sequential reading I get about 130 MBytes/s. The single drive benchmarks from Tom’s hardware were about 90 MBytes/s. So for sequential reads there is a (130-90)/90 or 44% increase over one disk. When reading over the network through SMB read performance is about 105 MB/s when copying to the disk of a remote server. So for us that means that either SMB adds an overhead of about 25 MB/s, we are hitting a network bottleneck, or that there is a write bottleneck on the destination server. This NAS has the ability to bond the interfaces so down the road I plan to implement this as see if there is any performance improvement.

Actual Write Performance:
Measuring writes with iostat doesn’t seem to work as well because I believe the writes are cached and serialized by the md device. Therefore the numbers jump around a lot. However I believe dd with a large block size and a timed write provides a reasonable benchmark:

[root@sonas1 test]# time dd of=testfile if=/dev/zero bs=1M count=100000008008+0 records in
8007+0 records out
real    2m48.763s
....
[root@sonas1 test]# echo $((8007/168))
47

So for writes we see 47MBytes/s. I tested this again with 64K block size and did see an improvement of 63MBytes/s. According to Tom’s hardware a single drive should put through about 90 MBytes/s for writes. So in this there is a (90-63)/90 or 30% decrease in sequential write performance for the 4 disk RAID 10 array in comparison to a single disk.

Oh Noes! My Theory didn’t hold up!
So there are a few possibilities here:

• My benchmarking of sequential performance is flawed.
• There is a bottleneck I am missing.
• Something is wrong with the configuration.
• My theory was wrong and I need to change it.

The first thing I did was research my benchmarking methodology of using dd to test sequential performance. I found that some people don’t recommend it. However, I used different block sizes which is one of the main concerns. Also dd writes sequentially and in our case that is what this device is used for. I was also careful to always flush my buffers before each test. Therefore I am happy with my benchmark to provide a reasonable ballpark estimate of performance for our intended workload.

For a bottleneck the two things I considered were CPU and the bus speed. According to the infallible Wikipedia 3 Gb/s SATA goes up to 286.10 MByte/s so I don’t believe that is a bottleneck. I watched CPU load during these tests and there was no significant load on either core.

For the possibility of something being wrong with the configuration I would love to hear if anyone has any recommendations. However, this is an appliance at heart so we did all the building through the Web GUI so I am not sure tweaking it is the best idea. One possibility is that there is a partition alignment issue with the these disks which have 4096 byte sectors. I am going to investigate this on my own and have also posted a question on Server Fault in hopes that someone in the community might already have the expert knowledge needed to diagnose if this is an issue.

That leaves the possibility that my theoretical expectations were wrong which seems to be the most likely. For the writes the extra 44% seems to fall short of what I would expect. I would have guessed that there would be about a 80% increase over a single disk. The idea that the write penalty due to mirroring would be cancelled out by the striping also turned out not to be true. The 30% hit also seems like a bigger hit than I would have expected.

The only real numbers of RAID 10 performance relative to a single disk that I could find were in the zdnet article Comprehensive Raid Performance Report. In the graphs comparing RAID 10 of 4 drives to the performance of a single drive I see a slight increase of write performance and a 100% increase in writes. However, it is not clear to me if this is random activity or sequential. I would expect that RAID might be a bigger advantage in random IO because of seek time.

Conclusion:
In the end this turns out to be a lot faster than our previous NAS and backups that took 25 minutes now take 5 minutes so we are happy with the upgrade. So I say farewell to our old NAS which had been given our love:

When it comes to predicting RAID performance based off of a single drive I would love to see more data for both sequential and random patterns:

• How do different drives and RAID cards perform?
• What is the formula of increase for RAID 10 and RAID 6 when you go from 4 drives to 6, 6 to 8, etc?
• How do different RAID levels actually compare in terms of single drive performance?

I think more comprehensive data could lead to betters formulas for estimating RAID performance. If these become available publicly on the net it could really help out system administration and turn RAID choice into a little bit less of a guessing game when it comes to performance.

In this post I am going to explore how extended iostat statistics can be useful to a system administrator beyond a binary “Disk is bottleneck / Disk is not bottleneck.” Before we can get to any of that however, we must make sure we have a basic background knowledge of the Disk IO Subsystem.

Linux Disk IO Subsystem:

I am not a kernel hacker, so this overview might be flawed in parts but hopefully it is accurate enough to give the background needed for analyzing the output of iostat.

Layer	Unit	Typical Unit Size
User Space System Calls	read() , write()
Virtual File System Switch (VFS)	Block	4096 Bytes
Disk Caches	Page
Filesystem (For example ext3)	Blocks	4096 Bytes (Can be set at FS creation)
Generic Block Layer	Page Frames / Block IO Operations (bio)
I/O Scheduler Layer	bios per block device (Which this layer may combine)
Block Device Driver	Segment	512 Bytes
Hard Disk	Sector	512 Bytes

There are two basic system calls, read() and write(), that a user process can make to read data from a file system. In the kernel these are handled by the Linux Virtual Filesystem Switch (VFS). VFS is an abstraction to all file systems so they look the same to the user space and it also handles the interface between the file system and the block device layer. The caching layer provides caching of disk reads and writes in terms of memory pages. The generic block layer breaks down IO operations that might involve many different non-contiguous blocks into multiple IO operations. The I/O scheduling layer takes these IO operations and schedules them based on order on disk, priority, and/or direction. Lastly, the device driver handles interfacing with the hardware for the actual operations in terms of disk sectors which are usually 512 bytes.

A Little Bit on Page Caching:

The page cache caches pages of data that do or will reside on disk. Therefore before it writes data to disk it puts it in memory, and before it reads data from disk it checks to see if it is in memory already (With the exception of Direct IO). Writing pages out to disk actually gets deferred. This is done to increase performance so writes can be grouped together more efficiently. When a page of disk data gets changed and needs to be written out to disk it is called “dirty”. Since it is dangerous to keep pages in memory for too long in case of a system shutdown the kernel’s pdflush threads scan for dirty pages and then flushes them out to disk. Linux will actually try to use as much memory as it can for caching files which is why the top command usually shows so much used memory. When you want to see how much memory is free for processes you can run the free command and look at the ‘-/+ buffers/cache’.

iostat output:

So with this background lets look at some of the output of iostat and tie it together with our background knowledge. Iostat can break down the statistics at both the partition level and then device level, however in this post I am going to focus on the device level.

The Overview Statistics: “Is it Saturated or Not?”

From iostat there are two summary statistics which are Input/Output CPU wait time (iowait) and device utilization which are both expressed in terms of percentages.

iowait is from the CPU’s perspective and it is the percentage of time that the CPU spent waiting for a IO device to be ready. Another way to look at iowait is the amount of time that the CPU could have been doing something but couldn’t because all the processes were waiting on the disk or the network devices.

Device utilization is covered throughly by Alex Gorbahev in Basic I/O Monitoring on Linux. He summarizes it as “The percentage of time the device spent servicing requests as opposed to being idle.”

iostat and caching:

It is import to note that iostat shows requests to the device (or partition) and not read and write requests from user space. So in the table above iostat is reading below the disk cache layer. Therefore, iostat says noting about your cache hit ratio for block devices. So it is possible that disk IO problems might be able to be resolved by memory upgrades. From my research there is no way to pull out a cache hit/miss ratio out of Linux easily when it comes to block devices which is a bit disappointing.  One suggestion from serverfault is to install a kernel with debuging symbols and use SystemTap to trace the VFS events and tie them together with the block layer events. I intend to explore this but I would prefer to see a way to get this data from /proc or /sys.

iostat Output for Random and Sequential Reads:

One of the main things to do when examining disk IO is to determine if the disk access patterns are sequential or random. This information can aid in our disk choices. When operations are random the seek time of the disk becomes more important. This is because physically the drive head has to jump around. Seek time is the measurement of the speed at which the heads can do this. For small random reads solid state disks can be a huge advantage.

So in fio I have created two different simple tests to run. The first is sequential reading, and the second is random reading. During these tests I ran iostat -x 3 throughout the test.

Snapshot of Random Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  172.67    0.00  1381.33     0.00     8.00     0.99    5.76   5.76  99.47

Snapshot of Sequential Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              13.00     0.00  367.00    0.00 151893.33     0.00   413.88     2.46    6.71   2.72 100.00

What is more important to me for this is not just what these numbers are but what, in the context of of random vs sequential reading and in context of the IO subsystem, they mean.

The first two columns, rrqm/s and wrqm/s, are read and write requests merged per second. In my above diagram of the Linux Block IO subsystem above I mentioned that that the scheduler can combine operations. This can be done when multiple operations are physically adjacent to each other on the device. So in sequential operation it would make sense to often see a large number of merges. In the snapshot of the random reads, we see no merges. However, the merging layer feels a little bit like “magic” and I don’t believe it is the best indicator of if the patterns are random or sequential.

The next 5 columns are read and write requests to the device (r/s, w/s), followed by the amount of sectors read and written from the device (rsec/s, wsec/s), and then the size of each request (avgrq-sz). In the random test there are 172 reads that result in 1,381 sectors being read in. In the sequential test there are 367 read request to 151,893 sectors being read. So in the random test we get about 8 sectors per request and in the sequential test we get 413 sectors per read. If you look closely, this happens to be the same number as avgrq-sz which does this math for us (Sectors Read / Read Operations). However it is worth noting that this is how it is calculated as the average request size does not differentiate between reads and writes. From these tests a low sector write/read to request ratio or small request sizes seem to indicate a random IO profile. I believe this to be a better indicator than the amount of merges as to whether or not there is random or sequential disk patterns.

The final 4 columns are the average queue length of requests to the device (avgqu-sz), how long requests took to be serviced including their time in the queue (await), how long requests took to be serviced by the device after they left the queue (svctm), and lastly the utilization percentage which I already mentioned in the overview statistics section. In the above example random requests take longer for the disk to service as expected because of the seek time. However, the queue itself ends up being shorter which I am unable to explain. Utilization, in more detail, is the service time in ms * total IO operations / 1000 ms. This gives the percentage of how busy the single disk was during the given time slice. I believe for a given utilization level a higher number of operations is probably indicative of a sequential pattern.

I have run various variations on the above. They include a mixture of reads and writes for both random and sequential data as well as sequential and random writes. For the writes I got similar results as far as the ratios were concerned and queue and services time were higher.

In the end it seems average request size is the key to show if the disk usage patterns are random or not since this is post merging. Taking this into the context of the layers above this might not mirror what an application is doing. This is because a read or write operations coming from user space might operate on a fragmented file in which case the generic block layer will break it up and it appears as random disk activity.

Conclusion:

As far as I am concerned this is only a start in interpreting IO statistics. I think these tests need to be repeated, perhaps with different tools to generate the disk IO, as my interpretations might just be totally off. Also, a pretty big limitation of what I did is that my work was all on a single disk and these numbers might have different results under various RAID configurations. I feel the inability to measure the cache hit ratio of reads on a block device is a significant shortcoming that I would love to see addressed since from a system administrators perspective the solution to certain IO problems might be to throw more memory at the problem.

Lastly, I want to make a point about these sort of low level statistics in general. Everything needs to monitored from the an application perspective as well. These statistics can be misleading and are most useful when they can be correlated with the data that actually matters to users of the applications, for example, response time from the user perspective. These also need to be monitored over time because you want to be able to see changes for capacity planning as well as to give them context to past performance when problems arise.

Further Reading:
http://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/
Understanding The Linux Kernel, Chapter 14
http://www.ufsdump.org/papers/io-tuning.pdf
http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/
http://www.kernel.org/doc/Documentation/iostats.txt

As system administrators we usually end up spending a good amount of time and thought on monitoring the systems we are responsible for. In this post I am not going to talk about how to monitor. Everyone talks about how to monitor, and what tools — Nagios, Munin, Monit, and so on. I think people tend to focus on that because it is often the more complex topic. However, I want to talk about what to monitor.

Honestly, I have never put much organized thought into what to monitor. I have always just tried to think of what is important and then make sure I monitor those things from both the outside and inside of my environment. This has served me fine in the past but this time around I want to do monitoring better. I have read a good amount of computer books and I am sure there are some that write about what to monitor. However, I want to start to look outside my direct field for inspiration. Most of the admins I have read about online monitor websites, email, and that sort of thing. But those are just websites and just email, and when trying to think of people who really need to get it right one thing popped right into my mind. The people who monitor nuclear power plants — I bet they feel the need to get monitoring right.

My research into this led me to the topic of reliability engineering and that led me to the topic of fault tree analysis. I combined those two and came up with the Fault Tree Handbook published by the office of Nuclear Regulatory Research of the U.S. Nuclear Regulatory Commission. It is a bit old, 1981 to be exact, but I figure these people take their monitoring seriously.

On Systems:

Before I get to what a fault tree is I am going to cover some of the highlights from the Fault Tree Handbook that I found relevant to monitoring and system administration. The first is the authors’ definition of a system:

A system is a deterministic entity comprising an interacting collection of discrete elements.

I think this definition has some interesting implications that the authors point out. The first is that we are interested in discrete elements. These can be parts, or systems themselves (subsystems). Identifying these parts and subsystems is our task when we decide what to monitor. Two primary things must be decided, what is the “external boundary” of the system and what will be the “limit of resolution”. For example with the external boundary, should I be monitoring seismic activity, or is that out of scope? The limit of resolution is simply how much detail do I need.  Do I need to monitor individual hard drives or just logical arrays.

The second interesting piece of this definition is that an “interacting collection” creates something that is more than the sum of its parts because the interactions must also be taken into account. These interactions can prove to be the most difficult to monitor, particularly in terms of performance, and is something I want to investigate more in the future.

Inductive and Deductive Approaches:

Inductive reasoning is reasoning from the specific to the general. Deductive logic goes the other way and is from the general to the specific. This is relevant when we try to think of what to monitor from a system failure perspective. If we take the deductive approach, we think of the most general case of failure, and then perhaps move onto the failure of a subsystem. From there, we try to think of what could cause that subsystem to fail. With inductive reasoning, we try to enumerate all the discrete items in the system and ask ourselves what happens if one or more pieces in the system fails. Both of these are valid and useful approaches, but fault trees are based in deductive reasoning. So on to what fault trees actually are.

Fault Trees:

A fault tree in its most basic form is a diagram that has a undesirable event at the root of the tree. Further down the tree is other event(s) that can lead up to that undesirable event. After reading the Fault Tree Handbook I went ahead and made my own “serverfault tree” with Visio (Larger Version):

The colors are meaningless and what actually has meaning is the shapes. A rectangle is an event that will be expanded upon. The circle is a basic event so it ends up being the limit of resolution of our fault tree. The diamond is an event that could be expanded upon but just isn’t for whatever reason. Lastly in this diagram there are those connector shapes. The inverted U shape is called an AND gate and the inverted V is an inclusive OR gate. What they mean is that all the items below them must either be OR’d or AND’d together to create the above event. So for example, if you look at the lowest leafs in my tree, you see there is an AND gate for the first and second hard drive basic events. This is because for RAID 5 two drives must fail for the array to fail. If we go one level higher in the tree there is an OR gate because with this particular server either the controller or the array (or both) must fail in order to create a failure of the disk subsystem.

Earlier I said that this tree is deductive because it is built from the top of the tree down. The tree is built from the most general case of a failure to the specific events that can cause it. However when building the tree I found an inductive approach helps as well. When I built mine I also thought of individual components and how they might effect the functioning of the server itself.

Why Bother?

In monitoring there are usually two general functions. One is to alert when something fails and record it. The other is to gather data about performance over time and record a history of that data which is particularly useful for capacity planning and trending. The fault tree is mostly useful when it comes to the component failure aspect. It highlights what must or ideally needs to be monitored. I think it also elegantly displays why these things must be monitored. The OR gates often imply that these things need to be monitored because they can cause failure. The AND gates show that monitoring this level can prevent failure because when one item is gone it should be fixed to reestablish redundancy.

When caught up in the details of configuring something as complex as Nagios it is understandable that the mind focuses. One might get so caught up in setting up the monitoring of a RAID array and configuring SNMP properly that the controller is completely overlooked. In the act of creating a fault tree we can pull ourselves away from the implementation details as system administrators which will allow us to brainstorm more creatively and effectively about what needs to be monitored. We can then go back and implement it.

My example is just a single server. If you run whole clusters you can build a separate tree, or extend one like my example, to cover the entire cluster.

If you noticed this form also shows where you do and don’t have redunancy. In general an AND gate will probably mean that there is some redundancy built into this particular part of the infrastructure. So you then have a picture of where you might need redundancy within the entire cluster if you build a fault tree or fault trees that represent the whole infrastructure.

I have set up complete network redundancy in the past using a combination of NIC teaming, multiple switches, HSRP, and BGP. When I tried to test that I got network redundancy right I took an inductive approach and simulated how redundancy worked when any of the particular pieces I could think of failed. This technique could be useful in developing test plans and analyzing redundancy that has been put into place. Also the implementation of this is relatively complex so again when an administrator is caught up in implementing it a flaw might be over looked in the bigger picture.

If you want to take this even further you can assign probabilities and possibly recovery time weights to each basic component to see where redundancy could best be implemented if you have a big enough organization to make this worth the time.

These could also be useful tools in troubleshooting just as monitoring itself is. When something has gone wrong and the pressure is on the fault tree can be used to see if what went wrong is an item in the fault tree. If it turns out the problem was not in the tree the item can be added to the tree for future implementation to help ensure that the mistake is not repeated. 

I see many benefits to the fault tree as a system administrator’s tool and I am adding it to my toolbox.

When the scope of the Stack Overflow engine was drastically increased, it became obvious that more hardware would be needed and that we would now have to put more emphasis on scaling.

Currently, everything lives at PEAK Internet in Corvallis, Oregon, which has worked out very well so far.  However, after much discussion, we came to the conclusion that we’ll be able to serve our globally distributed user base better if we at least move our heavier services to the East coast.  Since Fog Creek has had great experience with Peer 1, it seemed like a reasonable enough place to set up camp.  

We’ve decided to jump into two full-sized cabinets, each one with dual 40 amp power circuits pre-installed.  The current line of thinking is that we’ll spread our systems out evenly so that a serious disruption in one cabinet won’t take down service in its sister.  We want to be able to lose as many servers as possible and still be able to provide answers to you all.  How this actually works out in practice will be seen and I’m sure many revisions will follow as we learn.  

Each rack will contain two Rariton Dominion PX PDUs, one per power circuit.  These guys have a solid reputation and each unit gives us 20 receptacles for our servers: much better than the usual 8-10 receptacles that are often seen.  These units are manageable via SNMP, which allows us to monitor for spikes or other changes in activity over time.         

For basic data services, we’ll put two Dell PowerConnect 5448’s in each cabinet.  These are CLI-managed gigabit switches running a near-clone of Cisco’s IOS.  They’re great: you get basic trunking, VLAN, and SNMP support at a killer price.  Plus you can manage all of this by just ssh’ing into the switch itself.  Each switch will be interconnected with the rest (we’re protected from switching loops via STP, of course), and each server will have a connection to both switches within its parent rack.  

Now for the server hardware.  With only a few exceptions, our basic building block is the Dell R610.  These are amazingly flexible 1u boxes that are remarkably power efficient and capable of taking on almost any role assigned.   They also come loaded with the DRAC, a fantastic out-of-band management system that has saved me too many times to count.  We’ll be using these all over the place.

We’re going to run our own firewall / gateway layer on two Dell R610’s with an E5640 processor, 250GB of disk space at RAID 1, 8GB of RAM, and Intel NICs.  They’ll likely be running the OpenBSD operating system, which gives us PF for the firewall and CARP for high availability.     

There will be two load balancers built with the same hardware configuration as above, but running on top of FreeBSD.  It has been my experience that while OpenBSD is excellent for building firewalls, it hasn’t been road tested enough in other areas.  FreeBSD on the other hand, certainly has.  We’ll be using CARP again for high availability, and will run some combination of HAProxy and nginx to take care of the actual load balancing.

We’re expecting to deploy 10 web servers to start with (we currently run 6), all on the R610 platform and running Windows Server 2008 R2 Standard Edition.  They’ll have one Intel E5640, 250GB of disk space at RAID 1, and 16GB of RAM.

Two VMWare ESX servers will be built, the major difference being that these will have about a terabyte of usable storage at RAID 6 and 32GB of RAM.  

Two domain controllers will be in place, providing basic Active Directory services for all of this stuff.  They’ll be running on the 250GB of hard drive storage and 2 GB of RAM.

  

A syslog / monitoring server will run on a terabyte of storage at RAID 6 and 8GB of RAM.  All *nix syslog and Windows Eventlog data will be shipped over there for regular analysis and alerting if necessary.  It will be running FreeBSD.

A dedicated backup server will be in place to ensure that critical data is well protected.  We’re going to put this on an R610 with a 1.5TB of disk space and 16GB of RAM.  We’re mostly concerned about database backups, and this should give us considerable amounts of onsite storage.

The exceptions to the R610 rule are the database servers.  We’re going to put two in place, each on a Dell R710.  This shares the same guts as the R610, but is a 2u model and can house eight 2.5″ disks as compared to the six that the R610 can.  We’re going to load these up 64GB of RAM, a terabyte of storage at RAID 10, and are going to put two E5680 processors on board.

It should be understood that the above is not carved in stone: much of this will change during the actual deployment.  It’s great to lay out these big plans – they help light the way and keep you moving – but we’re not bound to any of it.  I tend to implement things organically, trying to build things in context of their environment.  What works, works. 

We’ll be posting more as things develop.

I began working with the Stack Overflow team at the beginning of May, and one of the first challenges I faced was to find a method to stop abusive users and bots from hammering the site.  While we’re happy to make way for legitimate search engines and users of our API, we need to ensure that the majority of our cpu cycles are going towards helping everyone find answers to their questions.  We needed a rate-limiting mechanism.  

After discovering that our tried-and-true load balancer did not have a native facility for rate limiting, I turned my attention towards nginx.  A quick scan of the documentation revealed that the limit_req_zone declaration should take care of things.  After a few tests with the development team, it was determined that it might actually work.  When any user surpassed the configured rate limit, a custom 503 error page would then be displayed.

Since HAProxy is already an important part of our stack, it didn’t make sense to rip it out.  Instead, I opted to slide nginx in front of haproxy, ultimately allowing us to add rate limiting to Stack Overflow with minimal effort and less risk to everyone.  If worse came to worse, nginx could easily be pulled back out of the stack without too much interruption.  I chose to break the work up into two phases: nginx without load balancing, and nginx with load balancing. 

I made the initial deployment on Sunday, May 16, and as you can see here, there were a few small glitches as our Monday morning traffic began to flow in.  Thankfully, Geoff’s quick thinking helped me see that we had exceeded the maximum amount of connections that the Linux kernel was configured to track, and I was able to adjust things appropriately.  After that, we had a few small bouts of slowness, which were overcome simply by increasing nginx’s worker process count and reloading the config.  I left this in place for a week, making extra sure that the new layer wasn’t going to have any adverse effects.

Rate limiting was then officially enabled on Sunday, May 23.  There was a bit of shakiness at first, as I did not enable the ‘nodelay’ feature.  This caused nginx to actually try and slow down offenders instead of just handing them 503s.  Once this was discovered, the feature was turned on and we were in business.

Over the past few days, I’ve been pouring over our http access logs, trying to determine if we have seen any amount of success.  I’m happy to say that we have: where I normally would see dozens of bots hitting us somewhere between 16,000 and 40,000 times an hour, I saw… none.  Victory for nginx! 

More to follow as we continue to tweak nginx and employ other systems to help us keep the channels clean…