Server Fault now has it’s own meta site at meta.serverfault.com. The idea of the meta site is that it is a place for people who participate on Server Fault to talk about the functioning of the site and the community itself. In the past if you wanted to talk about Server Fault you could do it on Stack Overflow’s meta but in general our system administration specific concerns were quickly buried by those of the larger original Stack Overflow site for programmers. If you are curious about the details of these per-site metas you can read more on the Stack Overflow blog.

I am excited about this new site because I believe as it gains traction it is going to help the community define itself. Server Fault has grown over the past year and has grown to a point where it is ready for the community to take ownership of itself. There has already been a question about how Server Fault appeals to professionals vs. administrators who are just starting out. This new place provides an opportunity for the dedicated members of the community to have conversations they feel address our specific concerns in creating an even better site for professional system administrators to get answers to their questions, become better system administrators, and form a great system administration community.

HSRP is not for WANs

Kyle Brandt

HSRP or Hot Standby Router Protocol is a protocol designed to provide a virtual gateway for the purposes of fail over. The basic idea is that you have the gateway IP address shared between two or more devices and only one of those devices holds it at a time. If one of those devices fails another device will take hold of this IP address which is commonly referred to as the virtual IP (VIP). Other implementations of this are CARP and VRRP but for this discussion I will just use HSRP.

This solution is great to provide a redundant gateway for hosts on the LAN side of your network which is what it was designed for. RFC 2281 defines HSRP and is pretty clear about what it was designed for:

“Using HSRP, a set of routers work in concert to present the illusion of a single virtual router to the hosts on the LAN.”
“In particular, the protocol protects against the failure of the first hop router when the source host cannot learn the IP address of the first hop router dynamically. … HSRP is not intended as a replacement for existing dynamic router discovery mechanisms and those protocols should be used instead whenever possible.”

Despite this, I have had a couple of occasions where providers have tried to convince me that HSRP will provide the same level of redundancy as a dynamic routing protocol would for routers — in other words beyond the first hop. This setup turns out to be problematic.

The HSRP Setup and the Unemployed Network Administrator:

The setup for full network redundancy that the provider gives to the client’s network administrator has three different instances of HSRP. The first is a virtual gateway for the client from the provider (192.168.0.1). In addition to this the provider has the client run HSRP on the client’s WAN side so the provider can route 172.16.0.0/24 via the VIP 192.168.0.5. The third instance is for the client LAN (172.16.0.1), which is how HSRP is actually supposed to be used.

At first glance this looks fine. The network administrator goes through the diagram and asks himself for every router “What happens if this router fails?”. If Provider 1 fails then Provider 2 picks up the 192.168.0.1 VIP and the layer 2 switches make sure that the packets go to the right ports. The opposite is true if Provider 1 fails. The same basic scenario happens if one of the client routers fails as well. So the network administrator thinks, “Great, it works, I just need to explain this to my boss and then I can go home….” He goes to explain this to his annoying pointy haired boss and after the boss asks “What about the blue boxes?”. The network administrators says, “Oh you must mean the switches”, however he thinks to himself “You are not smart enough to even know what they are called — why do I have to explain this?”

So he hypothetically takes out Switch 1 and explains what happens as fast as he can in hopes to confuse his boss enough that he just gets to home. “In this scenario Provider 2 picks up the VIP of 192.168.0.1 and Client 2 picks up the VIP of 192.168.0.5. The result is that there is still a path to the web server.” After this Mr. Pointy Hair says, “Okay that looks right to me but I do want you to go test it.” The network administrator grumbles and agrees to test it even though he knows it is going to work just fine because not only does he know it will work but the provider also agrees and they do this for a living.

So the network administrator sets it all up and pulls the plug on the switch only to discover that it just doesn’t work — why?! The reason is that although Client 2 changed the WAN VIP, the LAN VIP still remains on Client 1. Therefore replies go to Client 1 and then have no way to reach the gateway on Provider 2. The network administrator for the client calls the provider and starts complaining that he is going to look like an incompetent to his boss now. The provider says they will look into it and then calls back the next day and says, “Don’t worry, you can have your LAN VIP track the WAN VIP so they stay synced up — If the WAN goes down the LAN will follow”. The network administrator does this in test and shuts down Switch 1 and to his great relief everything works. They deploy this in production and live happily ever after…

Until one day switch 3 fails. When this happens the mirror problem of switch 1 failing happens. The web server can send packets out via Client 2 but replies end up stuck on Client 1 because although the LAN tracks the WAN, the WAN doesn’t track the LAN. This problem is found by the network consultant that Mr. Pointy Hair hired to find out what the hell happened. Soon after this Mr. Network Administrator gets canned.

So what is the Solution? You might say the solution is to have both interfaces on the client routers track each other. Personally I never tried this and it might even work. However at this point if you get this to work you really missed the moral of the story. The purpose of HSRP is clearly documented and it tells you to use a dynamic routing protocol such as BGP, OPSF, ISIS, or EIGRP. The people who created this are probably either smarter than you are or spent more time thinking about it. You don’t want to hack it for a WAN by adding extra layer 2 devices and various configuration tricks. Do what the smart people say, if your provider disagrees don’t stop until they give you a proper solution. They can do it no matter what they say, you just have to convince them.

For our datacenter in Corvallis we purchased a new Sans Digital EliteNAS EN104L+XR to replace our slower QNAP NAS.

In this post I want to try to figure out what the theoretical RAID performance of this NAS should be and then compare that to the performance we actually see.

I Present the EliteNAS:
The EliteNAS, which I would have called The l33t NAS, is a Network Attached Storage appliance built on top of GNU/Linux thats boasts the following:

  • 1U enclosure
  • Linux OS with custom web UI
  • Intel Xeon 3040 1.84GHz Dual Core CPU
  • dual gigabit ethernet
  • hot-swappable redundant power supplies
  • four SATA ports (software RAID)

We have installed 4×1.5TB drives into this configured in a RAID 10 configuration in hopes to see a significant gain in performance over our previous RAID 6 configuration.

This device also supports gigabit bonding. When they said that it is built on top of Linux, they mean it. All of the Linux bonding options are available through the GUI, Linux Software RAID is used, and you get access to the CLI via ssh if you are so inclined, which I must say I am:

[root@sonas1 ~]# /sbin/mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Thu Jul  8 03:25:55 2010
     Raid Level : raid10
     Array Size : 2930272128 (2794.53 GiB 3000.60 GB)
    Device Size : 1465136064 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
...
         Layout : near=2, far=1
     Chunk Size : 64K
...
    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

The file system used is XFS with the noatime and nodiratime mount options so there is not a write to the file every time it is read. It also comes with all the standard unix tools such as tcpdump and iostat.

As for the drives themselves we have 4 Western Digital Caviar Greens (WD15EARS). These are 3Gb/s SATA drives with a 64MB cache. Tom’s hardware has a nice review of these drives which I am going to rely on for individual drive performance. The highlights from the tests are (Ballpark Avg from Multiple Benchmark Tools):

Avg. IOPS: ~120
Avg. Sequential Throughput (Same for reads and writes): ~94 MBytes/s

What is RAID 10?
With Raid 10 some people might have discovered that there is RAID 1+0 and RAID 0+1 and you can find the difference between these talked about in this serverfault question. In this case we are using RAID 1+0:

So we have two sets of mirrors with the data striped over them. With this configuration we can lose up to two drives as long as both of those drives are not in same mirror.

What should our Theoretical Speeds of Raid 10 Be?
So I am lucky enough that Tom’s Hardware decided to go ahead and take care of the individual disk performance benchmarking as I already mentioned. So how do we figure out what happens in terms of performance when we combine them in RAID 10?

With RAID 10 the reads go over the stripe (RAID 0). So looking at our RAID 10 diagram we are reading from two RAID 1 mirrors. In this case we expect to see the sequential read performance of about two drives in RAID 0 according to the mdadm maintainer Niel Brown. If this was in the far layout configuration of RAID 10 we would get read performance on par with a 4 drive RAID 0 array but would incur a larger write penalty. For writes since we have a mirror every logical write causes two physical writes. However we have two sets of mirrors so I believe that we can expect the write performance of about a single disk. However as we will see from my benchmarks that turns out not to be the case.

Since the purpose of this device is for backups we are essentially interested in sequential disk activity because we will be streaming large contiguous files to and from the array.

Actual Read Performance:

Device:  rrqm/s    wrqm/s  r/s       w/s   rsec/s     wsec/s  rkB/s      wkB/s  avgrq-sz  avgqu-sz  await  svctm  %util
sda      7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sda1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sda2     7291.33   0.00    85.00     0.00  60362.67   0.00    30181.33   0.00   710.15    1.37      16.22  8.22   69.87
sdb      9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdb1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdb2     9092.00   0.00    88.00     0.00  73440.00   0.00    36720.00   0.00   834.55    1.33      15.18  7.52   66.13
sdc      5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdc1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdc2     5773.33   0.00    87.33     0.00  47552.00   0.00    23776.00   0.00   544.49    0.98      11.19  6.46   56.40
sdd      10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sdd1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sdd2     10592.00  0.00    103.33    0.00  85906.67   0.00    42953.33   0.00   831.35    1.05      10.19  5.34   55.20
sde      0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
sde1     0.00      0.00    0.00      0.00  0.00       0.00    0.00       0.00   0.00      0.00      0.00   0.00   0.00
md1      0.00      0.00    33109.33  0.00  264874.67  0.00    132437.33  0.00   8.00      0.00      0.00   0.00   0.00

For sequential reading I get about 130 MBytes/s. The single drive benchmarks from Tom’s hardware were about 90 MBytes/s. So for sequential reads there is a (130-90)/90 or 44% increase over one disk. When reading over the network through SMB read performance is about 105 MB/s when copying to the disk of a remote server. So for us that means that either SMB adds an overhead of about 25 MB/s, we are hitting a network bottleneck, or that there is a write bottleneck on the destination server. This NAS has the ability to bond the interfaces so down the road I plan to implement this as see if there is any performance improvement.

Actual Write Performance:
Measuring writes with iostat doesn’t seem to work as well because I believe the writes are cached and serialized by the md device. Therefore the numbers jump around a lot. However I believe dd with a large block size and a timed write provides a reasonable benchmark:

[root@sonas1 test]# time dd of=testfile if=/dev/zero bs=1M count=100000008008+0 records in
8007+0 records out
real    2m48.763s
....
[root@sonas1 test]# echo $((8007/168))
47

So for writes we see 47MBytes/s. I tested this again with 64K block size and did see an improvement of 63MBytes/s. According to Tom’s hardware a single drive should put through about 90 MBytes/s for writes. So in this there is a (90-63)/90 or 30% decrease in sequential write performance for the 4 disk RAID 10 array in comparison to a single disk.

Oh Noes! My Theory didn’t hold up!
So there are a few possibilities here:

  • My benchmarking of sequential performance is flawed.
  • There is a bottleneck I am missing.
  • Something is wrong with the configuration.
  • My theory was wrong and I need to change it.

The first thing I did was research my benchmarking methodology of using dd to test sequential performance. I found that some people don’t recommend it. However, I used different block sizes which is one of the main concerns. Also dd writes sequentially and in our case that is what this device is used for. I was also careful to always flush my buffers before each test. Therefore I am happy with my benchmark to provide a reasonable ballpark estimate of performance for our intended workload.

For a bottleneck the two things I considered were CPU and the bus speed. According to the infallible Wikipedia 3 Gb/s SATA goes up to 286.10 MByte/s so I don’t believe that is a bottleneck. I watched CPU load during these tests and there was no significant load on either core.

For the possibility of something being wrong with the configuration I would love to hear if anyone has any recommendations. However, this is an appliance at heart so we did all the building through the Web GUI so I am not sure tweaking it is the best idea. One possibility is that there is a partition alignment issue with the these disks which have 4096 byte sectors. I am going to investigate this on my own and have also posted a question on Server Fault in hopes that someone in the community might already have the expert knowledge needed to diagnose if this is an issue.

That leaves the possibility that my theoretical expectations were wrong which seems to be the most likely. For the writes the extra 44% seems to fall short of what I would expect. I would have guessed that there would be about a 80% increase over a single disk. The idea that the write penalty due to mirroring would be cancelled out by the striping also turned out not to be true. The 30% hit also seems like a bigger hit than I would have expected.

The only real numbers of RAID 10 performance relative to a single disk that I could find were in the zdnet article Comprehensive Raid Performance Report. In the graphs comparing RAID 10 of 4 drives to the performance of a single drive I see a slight increase of write performance and a 100% increase in writes. However, it is not clear to me if this is random activity or sequential. I would expect that RAID might be a bigger advantage in random IO because of seek time.

Conclusion:
In the end this turns out to be a lot faster than our previous NAS and backups that took 25 minutes now take 5 minutes so we are happy with the upgrade. So I say farewell to our old NAS which had been given our love:

When it comes to predicting RAID performance based off of a single drive I would love to see more data for both sequential and random patterns:

  • How do different drives and RAID cards perform?
  • What is the formula of increase for RAID 10 and RAID 6 when you go from 4 drives to 6, 6 to 8, etc?
  • How do different RAID levels actually compare in terms of single drive performance?

I think more comprehensive data could lead to betters formulas for estimating RAID performance. If these become available publicly on the net it could really help out system administration and turn RAID choice into a little bit less of a guessing game when it comes to performance.

In this post I am going to explore how extended iostat statistics can be useful to a system administrator beyond a binary “Disk is bottleneck / Disk is not bottleneck.” Before we can get to any of that however, we must make sure we have a basic background knowledge of the Disk IO Subsystem.

Linux Disk IO Subsystem:

I am not a kernel hacker, so this overview might be flawed in parts but hopefully it is accurate enough to give the background needed for analyzing the output of iostat.

Layer

Unit

Typical Unit Size

User Space System Calls

read() , write()


Virtual File System Switch (VFS)

Block

4096 Bytes

Disk Caches

Page


Filesystem (For example ext3)

Blocks

4096 Bytes (Can be set at FS creation)

Generic Block Layer

Page Frames / Block IO Operations (bio)


I/O Scheduler Layer

bios per block device (Which this layer may combine)


Block Device Driver

Segment

512 Bytes

Hard Disk

Sector

512 Bytes

There are two basic system calls, read() and write(), that a user process can make to read data from a file system. In the kernel these are handled by the Linux Virtual Filesystem Switch (VFS). VFS is an abstraction to all file systems so they look the same to the user space and it also handles the interface between the file system and the block device layer. The caching layer provides caching of disk reads and writes in terms of memory pages. The generic block layer breaks down IO operations that might involve many different non-contiguous blocks into multiple IO operations. The I/O scheduling layer takes these IO operations and schedules them based on order on disk, priority, and/or direction. Lastly, the device driver handles interfacing with the hardware for the actual operations in terms of disk sectors which are usually 512 bytes.

A Little Bit on Page Caching:

The page cache caches pages of data that do or will reside on disk. Therefore before it writes data to disk it puts it in memory, and before it reads data from disk it checks to see if it is in memory already (With the exception of Direct IO). Writing pages out to disk actually gets deferred. This is done to increase performance so writes can be grouped together more efficiently. When a page of disk data gets changed and needs to be written out to disk it is called “dirty”. Since it is dangerous to keep pages in memory for too long in case of a system shutdown the kernel’s pdflush threads scan for dirty pages and then flushes them out to disk. Linux will actually try to use as much memory as it can for caching files which is why the top command usually shows so much used memory. When you want to see how much memory is free for processes you can run the free command and look at the ‘-/+ buffers/cache’.

iostat output:

So with this background lets look at some of the output of iostat and tie it together with our background knowledge. Iostat can break down the statistics at both the partition level and then device level, however in this post I am going to focus on the device level.

The Overview Statistics: “Is it Saturated or Not?”

From iostat there are two summary statistics which are Input/Output CPU wait time (iowait) and device utilization which are both expressed in terms of percentages.

iowait is from the CPU’s perspective and it is the percentage of time that the CPU spent waiting for a IO device to be ready. Another way to look at iowait is the amount of time that the CPU could have been doing something but couldn’t because all the processes were waiting on the disk or the network devices.

Device utilization is covered throughly by Alex Gorbahev in Basic I/O Monitoring on Linux. He summarizes it as “The percentage of time the device spent servicing requests as opposed to being idle.”

iostat and caching:

It is import to note that iostat shows requests to the device (or partition) and not read and write requests from user space. So in the table above iostat is reading below the disk cache layer. Therefore, iostat says noting about your cache hit ratio for block devices. So it is possible that disk IO problems might be able to be resolved by memory upgrades. From my research there is no way to pull out a cache hit/miss ratio out of Linux easily when it comes to block devices which is a bit disappointing.  One suggestion from serverfault is to install a kernel with debuging symbols and use SystemTap to trace the VFS events and tie them together with the block layer events. I intend to explore this but I would prefer to see a way to get this data from /proc or /sys.

iostat Output for Random and Sequential Reads:

One of the main things to do when examining disk IO is to determine if the disk access patterns are sequential or random. This information can aid in our disk choices. When operations are random the seek time of the disk becomes more important. This is because physically the drive head has to jump around. Seek time is the measurement of the speed at which the heads can do this. For small random reads solid state disks can be a huge advantage.

So in fio I have created two different simple tests to run. The first is sequential reading, and the second is random reading. During these tests I ran iostat -x 3 throughout the test.

Snapshot of Random Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  172.67    0.00  1381.33     0.00     8.00     0.99    5.76   5.76  99.47

Snapshot of Sequential Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              13.00     0.00  367.00    0.00 151893.33     0.00   413.88     2.46    6.71   2.72 100.00

What is more important to me for this is not just what these numbers are but what, in the context of of random vs sequential reading and in context of the IO subsystem, they mean.

The first two columns, rrqm/s and wrqm/s, are read and write requests merged per second. In my above diagram of the Linux Block IO subsystem above I mentioned that that the scheduler can combine operations. This can be done when multiple operations are physically adjacent to each other on the device. So in sequential operation it would make sense to often see a large number of merges. In the snapshot of the random reads, we see no merges. However, the merging layer feels a little bit like “magic” and I don’t believe it is the best indicator of if the patterns are random or sequential.

The next 5 columns are read and write requests to the device (r/s, w/s), followed by the amount of sectors read and written from the device (rsec/s, wsec/s), and then the size of each request (avgrq-sz). In the random test there are 172 reads that result in 1,381 sectors being read in. In the sequential test there are 367 read request to 151,893 sectors being read. So in the random test we get about 8 sectors per request and in the sequential test we get 413 sectors per read. If you look closely, this happens to be the same number as avgrq-sz which does this math for us (Sectors Read / Read Operations). However it is worth noting that this is how it is calculated as the average request size does not differentiate between reads and writes. From these tests a low sector write/read to request ratio or small request sizes seem to indicate a random IO profile. I believe this to be a better indicator than the amount of merges as to whether or not there is random or sequential disk patterns.

The final 4 columns are the average queue length of requests to the device (avgqu-sz), how long requests took to be serviced including their time in the queue (await), how long requests took to be serviced by the device after they left the queue (svctm), and lastly the utilization percentage which I already mentioned in the overview statistics section. In the above example random requests take longer for the disk to service as expected because of the seek time. However, the queue itself ends up being shorter which I am unable to explain. Utilization, in more detail, is the service time in ms * total IO operations / 1000 ms. This gives the percentage of how busy the single disk was during the given time slice. I believe for a given utilization level a higher number of operations is probably indicative of a sequential pattern.

I have run various variations on the above. They include a mixture of reads and writes for both random and sequential data as well as sequential and random writes. For the writes I got similar results as far as the ratios were concerned and queue and services time were higher.

In the end it seems average request size is the key to show if the disk usage patterns are random or not since this is post merging. Taking this into the context of the layers above this might not mirror what an application is doing. This is because a read or write operations coming from user space might operate on a fragmented file in which case the generic block layer will break it up and it appears as random disk activity.

Conclusion:

As far as I am concerned this is only a start in interpreting IO statistics. I think these tests need to be repeated, perhaps with different tools to generate the disk IO, as my interpretations might just be totally off. Also, a pretty big limitation of what I did is that my work was all on a single disk and these numbers might have different results under various RAID configurations. I feel the inability to measure the cache hit ratio of reads on a block device is a significant shortcoming that I would love to see addressed since from a system administrators perspective the solution to certain IO problems might be to throw more memory at the problem.

Lastly, I want to make a point about these sort of low level statistics in general. Everything needs to monitored from the an application perspective as well. These statistics can be misleading and are most useful when they can be correlated with the data that actually matters to users of the applications, for example, response time from the user perspective. These also need to be monitored over time because you want to be able to see changes for capacity planning as well as to give them context to past performance when problems arise.

Further Reading:
http://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/
Understanding The Linux Kernel, Chapter 14
http://www.ufsdump.org/papers/io-tuning.pdf
http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/
http://www.kernel.org/doc/Documentation/iostats.txt