We recently changed the NICs in our web tier and primary database servers from Broadcom to Intel based NICs based on some … issues we had been having. After we put them in they worked reasonably well, but we knew that they could be faster and push more data. When I started to dig into just what we could do to tweak the pleathora of settings for the new NICs I found a few settings that would probably help a little bit, and one technology that could help us out tremendously.

Changes at a glance

• Turned on Intel I/OAT
• Adjusted Send and Recieve buffers to 2048 (max allowed)
• Turned off interrupt moderation
• Increased the Receive Side Scaling Queues from 1 to 4

The long version

The first and biggest change is that we turned on Intel’s I/OAT technology. It’s a collection of different techniques that work together to improve the performance of your host networking, as defined on Intel’s website:

• Intel® QuickData Technology — enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput.
• Direct Cache Access (DCA) — allows a capable I/O device, such as a network controller, to place data directly into CPU cache, reducing cache misses and improving application response times.
• Extended Message Signaled Interrupts (MSI-X) – distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance.
• Receive Side Coalescing (RSC) — aggregates packets from the same TCP/IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing.
• Low Latency Interrupts — tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency.

The catch is you need to be running a full Intel hardware stack. Your CPU, Motherboard, BIOS, NIC and OS all need to be compatible with the technology to be able to use it. Once you have the right stack in place, you might have to turn on a BIOS option, but that’s it no tweaking or poking to make it just right, it’s just right out of the box. Turning it on was as simple as flipping a BIOS setting, aptly named “Intel I/OAT.”

Performance tuning options

The performance options are all exposed via the PROSet utility making it nice and easy to change, no need to go digging into the registry for some esoteric key that may or may not be there. For each of them there is a trade off you need to consider. Some of the options will increase host CPU, some will cause higher host memory usage. To find the right value for your systems you really need to evaluate your overall situation and see if the trade offs are worth it.

Send and Receive Buffers

The Send and Receive buffers where set to the maximum allowed value of 2048. The trade off here is that you will consume more host memory. For us this is not a big deal since we have a lot of RAM on our boxes. Also, we had been seeing a good deal of Zero-window TCP packets when investigating our network so we needed to increase the buffer anyway.

Interrupt Moderation

The Interrupt Moderation feature was disabled. This feature allows you to have the NIC throttle the number of interrupts to the CPU which will limit the number of CPU cycles used by the NIC interrupts. Turning this off will increase your CPU usage, but it will also prevent packets from sitting there waiting for an interrupt to be proccessed. The increased cpu is a pain point for us right now (we are working on fixing that) but I believe it’s worth it.

Receive Side Scaling Queues

Receive Side Scaling is a technology that allows you to process a TCP connection across multiple cores. This allows for more efficient cache and processor usage when you TCP connection is not tied to a single core. When you are using this feature it will only use real physical cores to process TCP connections, you are not able to use this with hyper thread cores.

Receive Side Scaling Queues are essentially buffer space that is used between the NIC and the CPU when you are using Receive Side Scaling. This is another setting that has a trade off between host CPU and performance. I opted for the trade off, and increased the queues from 1 to 4 queues.

Additional notes

• Since I/OAT needs BIOS support and the BIOS on our web tier was woefully out of date anyway I updated to the latest BIOS for those machines
• I updated the Intel PROSet utility to verion 16.1 from 16.0

We are less than twenty four hours into using these new setting, but everything looks much much better through our peak traffic today. So far we are very happy with the results of these changes.