When it comes to system administration there are a two main questions that come up in relation to RAM:

• Which and how much RAM do I buy?
• How do I properly install it?

In order to be able to figure this out there are some fundamentals regarding RAM which a sysadmin has to know for servers. To be practical lets figure out what a particular stick of RAM is about (Which happens to be in our database servers):

> Registered ECC Dual Rank 8GB DDR3-1333 (PC3-10600)

Frequency and Bandwidth:

Wikipedia tells us that “frequency is the number of occurrences of a repeating event per a unit of time.” In the case of memory the “repeating event” we are concerned with is bits of data being sent from the cpu to RAM and the “unit of time” is a second. With memory this is done so many times we measure it in millions of times per second (aka megahertz or MHz). The bits are chunks of 64 bits which are often referred to as “words”. So the transfer frequency of our example RAM is 1333 MHz (the clock frequency is different, more on this later).

So how “fast” is this? In computing, speed is often measured in how much data you move in a certain amount of time (There is also latency). A common measurement is the amount of megabytes in a second. There are 8 bits per byte. We said before that this RAM can push 64 bits 1,333 million times per second. So 64 * 1,333,000,000 / 8 gives us 10,664,000,000 bytes a second, or roughly 10,600 MegaBytes per second. So we now have our bandwidth, 10,600 MB/s:

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

DDR3:

DDR stands for Double Data Rate. In the case of DDR memory the data transfer event actually manages to carry two chunks of data. With DDR2, you get 4, and with DDR3 you get 8. This does not include the rate of operations, but rather the amount of data chunks carried in each operation. What we end up having is different frequencies going on — clock ticks from the memory controller (which correspond to operations), bus and cpu frequencies, and finally an effective transfer frequency. These frequencies are all related to each other since this memory is synchronous. At the heart of all of this is the system clock, but memory is labeled by the transfer frequency because higher numbers sell better and it is probably the number most people would be interested in anyways. So the 10,600 MB/s throughput is based off of the transfer frequency and therefore takes DDR3 into account.

When it comes to “PC3” I am not entirely sure what that stands for, but DDR2 = PC2 and DDR3 = PC3. I am hoping a reader knows what it is for sure and can back it with some evidence.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Ranks and Capacity:

A rank is a independently accessible 64-bit area of memory. Today you buy memory in single, double, and quad ranks. To really get into ranks you will need to understand the basic signals used in ram which you can find in the “What every programmer should know about memory” in the referecences and then there is some detail on ranks in Addressing Increased Capacity Demand Using Commodity Memories. As a system administrator the main thing to understand is that there may be limitations on your motherboard such as the ability have mixed ranks and where certain ranks can go (On the R710, Quad-Ranks must go in the first DIMM slot of each channel).

The capacity is “8GB” — simply how much memory there is on the chip.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Registered and ECC

Registered RAM is also referred to as buffered RAM. It has a buffer between the memory controller and the memory chips. The end result is that you can have a higher capacity of RAM in the system but there is a little more latency. Memory you buy for servers will generally be registered memory. ECC stands for Error Correcting Code and includes parity on the memory to help ensure that the memory operation is valid.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Architecture, Channels, and the i7:

In the frequency and bandwidth section I mentioned that 64 bit chunks are being sent at a time. These chunks are being sent over a channel. When you have multiple channels the basic idea is that you are increasing your width — so dual channel is chunks of 128 bits. The current Intel i7 architecture supports 3 channels per processor (as in a chip you would hold in your hand). So in a dual processor system you have up to six channels. So with our simplified model we are talking rougly 30 GB/s of memory throughput per processor (10,600 * 3).

With recent architectures the memory was connected to the processor via the Front Side Bus (FSB). However with the i7 the memory controller is on the processors themselves and the memory is directly connected. There is also QPI (Quick Path Interconnect) which connects the PCI express bus to the processor. In multiple processor systems it also connects the processors together, so if Processor B wants Processor’s A memory it goes over the QPI.

There is a lot more to learn about memory. See the references section at the end of this post. Reading and writing about memory is difficult because there are a lot of differences between the general theory and what is actually happening so I and others skip over details. Reading the references or at least parts of them is probably needed to have a good picture of memory functions. You might also want to read about signal timings and interleaving, but I think the above covers the basics for a generalist system administrator.

Trade Offs — Channels, Capacity, and Speed:

So why does all this matter to a system administrator? The fundamental answer is that there are tradeoffs. Besides cost you have memory speed, error handling, and total capacity tradeoffs in the configuration you choose.

In our current database server, Dell R710, we are using the Intel 5600 series architecture. We have 2 processors which each have 3 channels of memory directly connected to each processor for a total of 6 channels. There are three general RAM population choices (Reference: Dell Poweredge 11g Whitepaper):

1. Optimize for speed (throughput). This configuration does not give the most RAM in the system in terms of capacity, but does give the fastest throughput possible for the memory. For the Dell R710, 4GB Registered DIMMs achieve this with a maximum amount of memory of 12 GB per CPU. This will let you run your memory at 1333 MHz.
2. A balance of speed and capacity. For this you would want dual-rank 1066 GB registered DIMMs. This allows for a capacity of 48GB per CPU but the memory can be clocked at 1033 MHz.
3. Optimize for capacity. With this option you would use the same chips as the balanced configuration but they get down clocked to 800 MHz for a total of 144 GB capacity.

According to Intel you get the best speed out of a balanced three channel configuration at this capacity. We just upgraded to 96 GB from 64 GB and have gone from a 2 channel configuration to a 3 channel configuration. This upgrade requires that the memory gets down clocked from DDR3-1333 to DDR3-1033. Although the clock speed has decreased for our memory, the number of channels or width has increased along with our capacity. Doing the math this in theory yields more memory throughput:

Two Channel:
(128 * 1,333,000,000) / 8 = 21,328 MB/s
Three Channel:
(192 * 1,033,000,000) / 8 = 24,792 MB/s

So our 3 channel configuration in our server is as follows:

So we match exactly option number 2 currently. Part of the reason you buy from a vendor like Dell is that they should be helping you get the ideal configuration, but knowing your RAM at a basic level will help make sure you are getting what you need.

Update:

George had informed me that he saw a memory speed 1333 MHz at boot. After looking at the output of CPU-Z and seeing that his was confirmed there as well I became confused as to how we could have 1333 MHz at a 96GB capacity:

After speaking with Dell I found that there was an information update regarding system memory for the R710. For 2R DIMMs they clock at 1333 not 1066 when you have 2 DIMMs per channel. So the proper calculation for our configuration is

(192 * 1,333,000,000) / 8 = 31,992 MB/s

This means that there was no trade off in clock speed when upgrading our capacity and to three channels.

References:

Kingston Technology memory ranking technical brief
What every programmer should know about memory — Highly Recommend, for those that want a lot of detail (PDF Version).
Memory technology evolution: an overview of system memory technologies, 8th ed.

The standard way to measure uptime is to measure the percent of time that your application is available over a period of time. This is always a good idea to do, but I don’t think it is the best way to measure the availability of your application for many companies. I think a better primary way is to think about how confident you are, your coworkers, and customers are in the availably of your application. If everyone is confident that your products will be up pretty much all the time then I think that this is a better measure than any “nines”.

I admit that going on how you feel as a primary measurement may seem a bit silly or touchy-feely. But I think your feelings and intuition solves a lot of issues with measuring uptime naturally. When even trying to figure out what a good uptime target is in percent of uptime some things you have to consider are:

Do you include maintenance time? This is probably more of an SLA issue than it is a practical one. For some sites, being down during off hours really isn’t a big deal. As an example, imagine an application that serves grade schoolers in the United States — it really doesn’t matter if it is down in the middle of the night.

Financial cost of achieving uptime vs cost of downtime? Some companies can map this pretty easily — “we loose X dollars per 30 seconds of downtime”. You then figure out the man power and equipment cost of high availability and all you need is some basic math skills. However, lots of companies are not finance or commerce companies so this is more difficult to measure. Take stackoverflow.com for example. We do have advertisers and our careers product, but our most valuable factor in my opinion is our users and our communities. How do you measure this other than keeping close ties to your users and see how they feel?

Development speed vs uptime? It seems intuitive that a fast development cycle will lead to more interruptions than a more careful and tested process. This might not be true, faster release cycles often mean smaller changes and might in the end cause less problems. I am not qualified to answer this, but it is another factor in uptime. If we accept this as true (I don’t, personally), then you have to weigh the cost of being down vs new features. The way this cost is measured is going to be difficult.

What counts as being down? If the site is slow, is this the same as being down? For example, at a certain point you might lose a customer to another commerce site if it is taking forever to load the purchase page. This is related to how Google now includes latency of your site in their page rank system. If you have new serious bugs but the site is still up, is this basically the same as being down?

Recovery Time: I think one of the most important things in uptime is recovery time. The primary thought most people have is “How do I keep from going down?”. Things go down no matter what, Facebook and Google have had some uptime issues before and everyone will. However, how much downtime you have when you do go down is a matter of how fast you can recover. I think recovery speed is a better place to start than attempting to eliminate going down at all. Of course, you need to do both.

The CHI of uptime: So if you are going to capture availability in a number it probably needs to include the above as factors in your equation as well as other things I probably haven’t even thought of. I heard a talk by Dharmesh Shah at Business of Software 2010 about measuring his customer’s happiness by factoring in various measurable stats as a Customer Happiness Index (CHI). (You can read about this talk here). Trying to do this with uptime would need to capture the above issues and is a worthy goal. However, getting there is not a small goal either.

Getting a good feeling for how confident your customers and employees are about the availability of your application captures all of these issues naturally. Uptime stats have their value, but I think numbers and stats are often overrated.