When it comes to system administration there are a two main questions that come up in relation to RAM:

• Which and how much RAM do I buy?
• How do I properly install it?

In order to be able to figure this out there are some fundamentals regarding RAM which a sysadmin has to know for servers. To be practical lets figure out what a particular stick of RAM is about (Which happens to be in our database servers):

> Registered ECC Dual Rank 8GB DDR3-1333 (PC3-10600)

Frequency and Bandwidth:

Wikipedia tells us that “frequency is the number of occurrences of a repeating event per a unit of time.” In the case of memory the “repeating event” we are concerned with is bits of data being sent from the cpu to RAM and the “unit of time” is a second. With memory this is done so many times we measure it in millions of times per second (aka megahertz or MHz). The bits are chunks of 64 bits which are often referred to as “words”. So the transfer frequency of our example RAM is 1333 MHz (the clock frequency is different, more on this later).

So how “fast” is this? In computing, speed is often measured in how much data you move in a certain amount of time (There is also latency). A common measurement is the amount of megabytes in a second. There are 8 bits per byte. We said before that this RAM can push 64 bits 1,333 million times per second. So 64 * 1,333,000,000 / 8 gives us 10,664,000,000 bytes a second, or roughly 10,600 MegaBytes per second. So we now have our bandwidth, 10,600 MB/s:

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

DDR3:

DDR stands for Double Data Rate. In the case of DDR memory the data transfer event actually manages to carry two chunks of data. With DDR2, you get 4, and with DDR3 you get 8. This does not include the rate of operations, but rather the amount of data chunks carried in each operation. What we end up having is different frequencies going on — clock ticks from the memory controller (which correspond to operations), bus and cpu frequencies, and finally an effective transfer frequency. These frequencies are all related to each other since this memory is synchronous. At the heart of all of this is the system clock, but memory is labeled by the transfer frequency because higher numbers sell better and it is probably the number most people would be interested in anyways. So the 10,600 MB/s throughput is based off of the transfer frequency and therefore takes DDR3 into account.

When it comes to “PC3” I am not entirely sure what that stands for, but DDR2 = PC2 and DDR3 = PC3. I am hoping a reader knows what it is for sure and can back it with some evidence.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Ranks and Capacity:

A rank is a independently accessible 64-bit area of memory. Today you buy memory in single, double, and quad ranks. To really get into ranks you will need to understand the basic signals used in ram which you can find in the “What every programmer should know about memory” in the referecences and then there is some detail on ranks in Addressing Increased Capacity Demand Using Commodity Memories. As a system administrator the main thing to understand is that there may be limitations on your motherboard such as the ability have mixed ranks and where certain ranks can go (On the R710, Quad-Ranks must go in the first DIMM slot of each channel).

The capacity is “8GB” — simply how much memory there is on the chip.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Registered and ECC

Registered RAM is also referred to as buffered RAM. It has a buffer between the memory controller and the memory chips. The end result is that you can have a higher capacity of RAM in the system but there is a little more latency. Memory you buy for servers will generally be registered memory. ECC stands for Error Correcting Code and includes parity on the memory to help ensure that the memory operation is valid.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Architecture, Channels, and the i7:

In the frequency and bandwidth section I mentioned that 64 bit chunks are being sent at a time. These chunks are being sent over a channel. When you have multiple channels the basic idea is that you are increasing your width — so dual channel is chunks of 128 bits. The current Intel i7 architecture supports 3 channels per processor (as in a chip you would hold in your hand). So in a dual processor system you have up to six channels. So with our simplified model we are talking rougly 30 GB/s of memory throughput per processor (10,600 * 3).

With recent architectures the memory was connected to the processor via the Front Side Bus (FSB). However with the i7 the memory controller is on the processors themselves and the memory is directly connected. There is also QPI (Quick Path Interconnect) which connects the PCI express bus to the processor. In multiple processor systems it also connects the processors together, so if Processor B wants Processor’s A memory it goes over the QPI.

There is a lot more to learn about memory. See the references section at the end of this post. Reading and writing about memory is difficult because there are a lot of differences between the general theory and what is actually happening so I and others skip over details. Reading the references or at least parts of them is probably needed to have a good picture of memory functions. You might also want to read about signal timings and interleaving, but I think the above covers the basics for a generalist system administrator.

Trade Offs — Channels, Capacity, and Speed:

So why does all this matter to a system administrator? The fundamental answer is that there are tradeoffs. Besides cost you have memory speed, error handling, and total capacity tradeoffs in the configuration you choose.

In our current database server, Dell R710, we are using the Intel 5600 series architecture. We have 2 processors which each have 3 channels of memory directly connected to each processor for a total of 6 channels. There are three general RAM population choices (Reference: Dell Poweredge 11g Whitepaper):

1. Optimize for speed (throughput). This configuration does not give the most RAM in the system in terms of capacity, but does give the fastest throughput possible for the memory. For the Dell R710, 4GB Registered DIMMs achieve this with a maximum amount of memory of 12 GB per CPU. This will let you run your memory at 1333 MHz.
2. A balance of speed and capacity. For this you would want dual-rank 1066 GB registered DIMMs. This allows for a capacity of 48GB per CPU but the memory can be clocked at 1033 MHz.
3. Optimize for capacity. With this option you would use the same chips as the balanced configuration but they get down clocked to 800 MHz for a total of 144 GB capacity.

According to Intel you get the best speed out of a balanced three channel configuration at this capacity. We just upgraded to 96 GB from 64 GB and have gone from a 2 channel configuration to a 3 channel configuration. This upgrade requires that the memory gets down clocked from DDR3-1333 to DDR3-1033. Although the clock speed has decreased for our memory, the number of channels or width has increased along with our capacity. Doing the math this in theory yields more memory throughput:

Two Channel:
(128 * 1,333,000,000) / 8 = 21,328 MB/s
Three Channel:
(192 * 1,033,000,000) / 8 = 24,792 MB/s

So our 3 channel configuration in our server is as follows:

So we match exactly option number 2 currently. Part of the reason you buy from a vendor like Dell is that they should be helping you get the ideal configuration, but knowing your RAM at a basic level will help make sure you are getting what you need.

Update:

George had informed me that he saw a memory speed 1333 MHz at boot. After looking at the output of CPU-Z and seeing that his was confirmed there as well I became confused as to how we could have 1333 MHz at a 96GB capacity:

After speaking with Dell I found that there was an information update regarding system memory for the R710. For 2R DIMMs they clock at 1333 not 1066 when you have 2 DIMMs per channel. So the proper calculation for our configuration is

(192 * 1,333,000,000) / 8 = 31,992 MB/s

This means that there was no trade off in clock speed when upgrading our capacity and to three channels.

References:

Kingston Technology memory ranking technical brief
What every programmer should know about memory — Highly Recommend, for those that want a lot of detail (PDF Version).
Memory technology evolution: an overview of system memory technologies, 8th ed.