Know Your RAM

Kyle Brandt

When it comes to system administration there are a two main questions that come up in relation to RAM:

  • Which and how much RAM do I buy?
  • How do I properly install it?

In order to be able to figure this out there are some fundamentals regarding RAM which a sysadmin has to know for servers. To be practical lets figure out what a particular stick of RAM is about (Which happens to be in our database servers):

> Registered ECC Dual Rank 8GB DDR3-1333 (PC3-10600)

Frequency and Bandwidth:

Wikipedia tells us that “frequency is the number of occurrences of a repeating event per a unit of time.” In the case of memory the “repeating event” we are concerned with is bits of data being sent from the cpu to RAM and the “unit of time” is a second. With memory this is done so many times we measure it in millions of times per second (aka megahertz or MHz). The bits are chunks of 64 bits which are often referred to as “words”. So the transfer frequency of our example RAM is 1333 MHz (the clock frequency is different, more on this later).

So how “fast” is this? In computing, speed is often measured in how much data you move in a certain amount of time (There is also latency). A common measurement is the amount of megabytes in a second. There are 8 bits per byte. We said before that this RAM can push 64 bits 1,333 million times per second. So 64 * 1,333,000,000 / 8 gives us 10,664,000,000 bytes a second, or roughly 10,600 MegaBytes per second. So we now have our bandwidth, 10,600 MB/s:

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

DDR3:

DDR stands for Double Data Rate. In the case of DDR memory the data transfer event actually manages to carry two chunks of data. With DDR2, you get 4, and with DDR3 you get 8. This does not include the rate of operations, but rather the amount of data chunks carried in each operation. What we end up having is different frequencies going on — clock ticks from the memory controller (which correspond to operations), bus and cpu frequencies, and finally an effective transfer frequency. These frequencies are all related to each other since this memory is synchronous. At the heart of all of this is the system clock, but memory is labeled by the transfer frequency because higher numbers sell better and it is probably the number most people would be interested in anyways. So the 10,600 MB/s throughput is based off of the transfer frequency and therefore takes DDR3 into account.

When it comes to “PC3″ I am not entirely sure what that stands for, but DDR2 = PC2 and DDR3 = PC3. I am hoping a reader knows what it is for sure and can back it with some evidence.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Ranks and Capacity:

A rank is a independently accessible 64-bit area of memory. Today you buy memory in single, double, and quad ranks. To really get into ranks you will need to understand the basic signals used in ram which you can find in the “What every programmer should know about memory” in the referecences and then there is some detail on ranks in Addressing Increased Capacity Demand Using Commodity Memories. As a system administrator the main thing to understand is that there may be limitations on your motherboard such as the ability have mixed ranks and where certain ranks can go (On the R710, Quad-Ranks must go in the first DIMM slot of each channel).

The capacity is “8GB” — simply how much memory there is on the chip.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Registered and ECC

Registered RAM is also referred to as buffered RAM. It has a buffer between the memory controller and the memory chips. The end result is that you can have a higher capacity of RAM in the system but there is a little more latency. Memory you buy for servers will generally be registered memory. ECC stands for Error Correcting Code and includes parity on the memory to help ensure that the memory operation is valid.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Architecture, Channels, and the i7:

In the frequency and bandwidth section I mentioned that 64 bit chunks are being sent at a time. These chunks are being sent over a channel. When you have multiple channels the basic idea is that you are increasing your width — so dual channel is chunks of 128 bits. The current Intel i7 architecture supports 3 channels per processor (as in a chip you would hold in your hand). So in a dual processor system you have up to six channels. So with our simplified model we are talking rougly 30 GB/s of memory throughput per processor (10,600 * 3).

With recent architectures the memory was connected to the processor via the Front Side Bus (FSB). However with the i7 the memory controller is on the processors themselves and the memory is directly connected. There is also QPI (Quick Path Interconnect) which connects the PCI express bus to the processor. In multiple processor systems it also connects the processors together, so if Processor B wants Processor’s A memory it goes over the QPI.

There is a lot more to learn about memory. See the references section at the end of this post. Reading and writing about memory is difficult because there are a lot of differences between the general theory and what is actually happening so I and others skip over details. Reading the references or at least parts of them is probably needed to have a good picture of memory functions. You might also want to read about signal timings and interleaving, but I think the above covers the basics for a generalist system administrator.

Trade Offs — Channels, Capacity, and Speed:

So why does all this matter to a system administrator? The fundamental answer is that there are tradeoffs. Besides cost you have memory speed, error handling, and total capacity tradeoffs in the configuration you choose.

In our current database server, Dell R710, we are using the Intel 5600 series architecture. We have 2 processors which each have 3 channels of memory directly connected to each processor for a total of 6 channels. There are three general RAM population choices (Reference: Dell Poweredge 11g Whitepaper):

  1. Optimize for speed (throughput). This configuration does not give the most RAM in the system in terms of capacity, but does give the fastest throughput possible for the memory. For the Dell R710, 4GB Registered DIMMs achieve this with a maximum amount of memory of 12 GB per CPU. This will let you run your memory at 1333 MHz.
  2. A balance of speed and capacity. For this you would want dual-rank 1066 GB registered DIMMs. This allows for a capacity of 48GB per CPU but the memory can be clocked at 1033 MHz.
  3. Optimize for capacity. With this option you would use the same chips as the balanced configuration but they get down clocked to 800 MHz for a total of 144 GB capacity.

According to Intel you get the best speed out of a balanced three channel configuration at this capacity. We just upgraded to 96 GB from 64 GB and have gone from a 2 channel configuration to a 3 channel configuration. This upgrade requires that the memory gets down clocked from DDR3-1333 to DDR3-1033. Although the clock speed has decreased for our memory, the number of channels or width has increased along with our capacity. Doing the math this in theory yields more memory throughput:

Two Channel:
(128 * 1,333,000,000) / 8 = 21,328 MB/s
Three Channel:
(192 * 1,033,000,000) / 8 = 24,792 MB/s
So our 3 channel configuration in our server is as follows:

So we match exactly option number 2 currently. Part of the reason you buy from a vendor like Dell is that they should be helping you get the ideal configuration, but knowing your RAM at a basic level will help make sure you are getting what you need.

Update:

George had informed me that he saw a memory speed 1333 MHz at boot. After looking at the output of CPU-Z and seeing that his was confirmed there as well I became confused as to how we could have 1333 MHz at a 96GB capacity:

After speaking with Dell I found that there was an information update regarding system memory for the R710. For 2R DIMMs they clock at 1333 not 1066 when you have 2 DIMMs per channel. So the proper calculation for our configuration is

(192 * 1,333,000,000) / 8 = 31,992 MB/s

This means that there was no trade off in clock speed when upgrading our capacity and to three channels.

References:

Kingston Technology memory ranking technical brief
What every programmer should know about memory — Highly Recommend, for those that want a lot of detail (PDF Version).
Memory technology evolution: an overview of system memory technologies, 8th ed.

«
»
  1. crb says:

    Hi Kyle,

    Great article!

    You’re missing some words in your numbered list under “Trade offs”: e.g. “You get to up to but still get to clock your at 1033 MHz”. That sentence no verb! :)

  2. Tweets that mention Know Your RAM - blog.serverfault.com -- Topsy.com says:

    [...] This post was mentioned on Twitter by Jonathan Angliss. Jonathan Angliss said: RT @GABeech: MMM Sweet sweet memory: http://blog.serverfault.com/post/know-your-ram/ [...]

  3. Kyle Brandt says:

    Thanks for catching that @crb — I jumped the gun on posting this — fixed that and several other typos.

  4. Travis says:

    Hmm, how about discussing latency & timing figures? Like: Timing: 8-8-8-24 CAS: 8

  5. Kyle Brandt says:

    Hi Travis,

    I considered that but in my experience it doesn’t come up in system administration often. Last time I looked at Dell and HP and I didn’t see options for timings so I didn’t really think it is a need-to-know for most sysadmins. That being said more knowledge is awesome and http://lwn.net/Articles/250967/ goes into the details of the various signals.

  6. Travis says:

    Great info there, thanks!

  7. Know Your RAM – blog.serverfault.com : : ram says:

    [...] Czytaj więcej: Know Your RAM – blog.serverfault.com [...]

  8. Matthew Bramer says:

    I really enjoyed this article… I never knew how to calculate the math for the clock settings. Actually, I’ve never looked it up :P

    I’m trying to follow the equation and I’m stumped as to why in a 3-channel config, you are multiplying by 2. Should it bump to three at that point? It’d be nice to have the formula spelled out for me. Any chance I can talk you into doing that?

    Cheers, Matt

  9. Can Dual Channel DDR Ram be used in a Computer that does not support Dual Channel DDR? | FireBall Tech Computer Repair - Tucson Computer Repair - Bill Arnoldi - Computer Repair Tucson - Rita Ranch Computer Repair says:

    [...] Understanding Types of RAMShould I buy an SLI motherboard or not? [western digital wd2500ks …Know Your RAM – blog.serverfault.comOther articles you might like;XCM XFPS Rateup Adapter for PlayStation 3 PS3NVIDIA Launches Tegra [...]

  10. Warren says:

    from the wikipedia article on DDR3 (http://en.wikipedia.org/wiki/DDR3_SDRAM) – “DDR3-xxx denotes data transfer rate, and describes raw DDR chips, whereas PC3-xxxx denotes theoretical bandwidth (with the last two digits truncated), and is used to describe assembled DIMMs. Bandwidth is calculated by taking transfers per second and multiplying by eight. This is because DDR3 memory modules transfer data on a bus that is 64 data bits wide, and since a byte comprises 8 bits, this equates to 8 bytes of data per transfer.”

  11. Kyle Brandt says:

    @Matthew I am not sure to which part you mean — can you quote the part where I am multiplying by 2 for the 3 channel configuration?

    I wouldn’t really call this a formula, rather just an understanding of transfer rates. So when we talk about the total effective transfer rate for all the channels:

    Q: What is being transfered? A: 64 bits per channel

    Q: What is the rate of transfer? A: The transfer frequency of the RAM, for example 1333 MHz. This is 1.3 billion times a second

    So… (64 bits * [# of Channels] * Ram Transfer Frequency = bits transfered per second And if you want bytes you divide by eight since there are 8 bits per byte.

  12. diams99 says:

    I guess the part is: Two Channel: (128 * 1,333,000,000) / 8 = 21,328 MB/s Three Channel: (192 * 1,033,000,000) / 8 = 24,792 MB/s

    …for I was trying to figure out the 192 by myself too. Couldn’t it be: Two Channel: (2 * 64 * 1,333,000,000) / 8 = 21,328 MB/s Three Channel: (3 * 96 * 1,033,000,000) / 8 = 29,750 MB/s

    ?

  13. Kyle Brandt says:

    Ah okay — I see the confusion. In this 64 is not the amount of RAM. 64 bits is the amount of data transfered per channel in a transfer. That transfer happens, for example, 1,333,000,000 times a second. If you have to channels, 128 bits gets transfered 1,333,000,000 times a second. 3 Channels is 192 bits transfered 1,333,000,000 times a second.

    So basically a channel is a 64 bit pipe. If you have 3 channels — you get 3 pipes.

  14. diams99 says:

    ah ok, makes sense. Thanks for the clarification.

  15. Sysadmin Sunday #15 « Boxed Ice Blog says:

    [...] Know Your RAM [...]

  16. toplinetech says:

    Cheapest memory on amazon! or well beat it! http://www.amazon.com/shops/toplinetech

  17. Right sizing ISV installations for the best bang for the buck! - Enterprise Applications on Microsoft SQL Server - Site Home - TechNet Blogs says:

    [...] There can be some limits on the maximum memory per core based on memory type, the number of memory channels for each core and other factors. The lower limit appears to be 12GB/core. For more information see Know Your RAM. [...]

  18. Oscar Cervantes says:

    Hi!

    greate article, but… how can I determine the configuration that I need?, I mean, the first one it’s helpful for reports or to many transactional servers… 

    do you know what I mean?

  19. Anonymous says:

    A nice level of detail, I wish more coders would think about memory consumption

  20. Johnsamuel says:

    http://www.memorystock.com/ 

Leave a Reply