Transfer rates and the number of packets you send are measured in units of a certain quantity of data per units of time. The unit of time that everyone is used to is the second. The standard quantity of data that is used in the networking field is bits and the standard time unit is seconds. So for example, the standard network interface these days is 1 Gigabit per second. So the quantity of data is a Gigabit, and the unit of time is a second. We call this the transfer rate. The key thing to remember is that this is a fixed ratio of data over time. Because of this, you can divide the ratio by any number you want to (Ignoring the complexities of the discrete properties of Ethernet frequencies, system clocking, etc). So, 500 Mbit over a half second is the same fixed ratio as 1 Gigabit per second.

The thing is though, in computing, a second is a really, really, really long time. This is important, because when we choose what unit of time to express this in, what we are doing is graph smoothing (It is sort of, although not really, like taking an average).

For example, we could transfer 900 Mbit in half of a second and another 100 Mbit for the other half of that second. How much data was transferred during that second? The answer is 1 Gbit. If we transfer 500 Mbit per half second and another 500 Mbit per the other half second — this is also 1 Gbit per second:. This effect is illustrated in these Megabits per half second graphs:

These two are clearly not the same thing, but when you express them as the amount of data transfered over a second they are. This is important because a 1 Gbit per second interface is also a 500 Mbit per half second interface — and a 500 Mbit per half second interface can’t transfer 900 Mbits per half second (I am ignoring any buffering effects, but in practice we have found this to be essentially true).

This effect is made even worse by most monitoring tools because most take samples every 5 minutes. So what you are really seeing is the transfer rate per 5 minutes converted to a per second rate. This sort of thing is why people say data can lie.

Why Should you Care?

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

We have bonded these interfaces using Intel Load Balancing (ALB/RLB) and for the most part our discards have gone away. We did this on all but one of our web servers for a while and found that the one that didn’t have the bonded interface had discards climbing while the others did not.

A second is a long time — be wary of trusting it too much to measure things.

«
»
  • Wow. I never considered a situation where the 1 second time period causes problems like this. Nice troubleshooting effort! Thanks for sharing these tiny details of running a high traffic network 🙂

    • Wow. Your sarcasm really adds a lot to the comment thread @robgolding63:disqus  Thanks for sharing your content-free thoughts. [NOT]

      Some readers (me) are visiting from English Language & Usage http://english.stackexchange.com 

      I like @openid-28206:disqus blog posts. Actually, I’m wondering if fixing that micro-burst problem from the prior post helped your network. Also wanted to mention that StackOverflow has THE BEST “We are offline” static page. Very descriptive. (Not 404, just down for maintenance). 

      P.S. Sorry Rob, my ire really should be targeted at Richard, but he is commenting as a guest so I couldn’t fine tune. After all, you did caveat with a 😉   Peace?

      • Actually, Ellie, no sarcasm was intended whatsoever. Re-reading my comment, I can’t even see how you came to the conclusion that I was ridiculing Kyle’s post. Please, reconsider your response.

        • Well, I totally misinterpreted that. 

          My apologies. I am so embarrassed. The burden of poor social skills. I’m so sorry.

  • And that is why logging levels etc have varying granularities: DEBUG, INFO, WARN, ERROR, etc.

    I’ve run into issues like this a couple times myself, and being able to explore deeper has been a lifesaver every time!

  • Dave Fiddes

    Have you guys looked into turning on Ethernet flow-control? If you have a uniform speed of 1Gbps everywhere then flow-control would allow your switches to smooth out these short impulse loads. If you have a mixed speed network beware though…traffic to a slow port can bring the rest of the network to a grinding halt with RX/TX flow-control enabled everywhere.

    • Hi Dave,

      According to the Cisco docs for our switches, the 2960S can not send PAUSE frames but only receive them. Since it seems to be the switches that are discarding — this would be the feature I would need I think. The fancy Nexus line does support this. This combined with bigger caches I believe is called “Delayed Drop”.

      I have a call scheduled with some Cisco engineers next week to discuss just this.

      Thank you, Kyle

      • Willy Tarreau

        Hi Kyle,

        I didn’t know that cisco did not support flow-control everywhere. Sometimes the workaround can be as simple as setting up a cheap switch in between, though it’s dirty !

        Cheers, Willy

  • There is something that I’m not getting. Lets suppose that I have the following setup:

    Computer Switch

    Where link is of 1Gbps. Now, given this setup, the computer can not physically receive more that 1Gb per second or whatever that works out to in milliseconds. Physically the link can’t deliver more.

    In the other direction (from the computer to the switch) again I’m not able to send more that 1Gbps physically.

    I only see two possibilities for packet loss here: – if the NIC driver is stupid and doesn’t block the sending application when the internal buffers are full (ie. it lets the application send more than the link speed) – if the computer is getting data from more than one source, in which case the packet dropping should be occurring in the switch

    What am I missing?

    • D J

      It’s not the links but the bit in between.

      Switches store and forward packets generally. Packets are held in fast memory attached to the switch fabric. If you have a lot of things going on within your switch then this intermediate buffer memory gets used up resulting in packet loss.

      You’re probably thinking: but I bought a wire speed switch! In my experience that is somewhat a variable term. If you have streams going from one port to another port and all streams are independent (i.e. port 1 talks to port 2, port 3 to port 4, etc) then you can achieve wire speed switching with cheap switches. If you have multiple network flows into a port then you get a small but significant delay that crosses the whole switch and causes buffer memory to be used up for a tiny period of time. For big enough bursts of 1Gbps traffic you can blow through a 8Mbit buffer pretty fast… Cheaper switches have a slower clocked switch fabric, slower buffer memory and less of it. It all makes it more likely to run out.

      Ethernet flow control smooths things out somewhat but essentially pushes stuff back up to the NIC that’s injecting the packet into the network. You can then use the buffering on the host (or router) to paper over the problem… Pushing that sort of  network to the extreme can get some weird and wonderful effects including crazy latencies. Not all packet loss is bad which is why flow control is generally turned off on enterprise switches.

      Some excellent resources on the subject if you’re interested:  – Network Algorithmics by George Varghese (http://www.elsevier.com/wps/find/bookdescription.cws_home/704109/description#description)  – Jim Gettys Bufferbloat crusade (http://www.bufferbloat.net/)

  • zippy

    Why use ALB/RLB instead of 802.3ad?

  • Richard

    This is a dumb argument. You’re asking people who have been measuring this for their entire career to change everything they know and hold to be true. It’s not a strong enough argument.

    • Actually, this is basic math coupled with what any good engineer should be aware of: the importance of accurate measurements.  If you don’t understand or care about either, then your entire career has been a joke.  No one else’s world has been radically shaken by this insightful, but hardly surprising, blog post.  “Better measurement gives better results” is virtually a tautology.

    • Hi Richard,

      Well the title was aimed to be a little catchy — being a blog post. Really what I am saying is understand what your measurement really is. In most cases Bandwidth per 5 minutes, converted a per second measurement is fine. However, in higher performance transactional workloads it can be problematic.

      The import thing is to understand the data and not take it for granted.

      -Kyle

    • You’re really arguing that “we’ve always done it that way” is a valid reason? 

  • Michael S. FIscher

    Does your equipment let you monitor the transmit/receive queue depths?  They are there specifically to queue up packets that are currently being received or transmitted too fast for the hardware to process.  If so, a high discard rate is a sure sign that bursts like this are resulting in discarded packets.

  • gjvc

    No shit.

  • I may be missing something here, but

    The NIC output buffer should never otherflow, as the driver can tell the OS to stop sending packets.

    If the switch is unable to process a packet, why does the switch not just fake a collision so the NIC resends the packet?

    • Anon

      “fake a collision”? The NIC itself resends packets? You have no idea of what you are talking about.

  • Liam N

    So, what does one do to monitor this in the long-term? (Short of running wireshark on every one of our servers..)

    • Liam: I am planning on scheduling TCP samples. See todays post on high resolution monitoring: http://blog.serverfault.com/post/the-three-monitoring-perspectives-part-1-ground-level/

      Basically you have to find what you think is a Representative sample. 

      Also, with micro bursts you should see drops on the  switch interfaces

      • Liam N

        Ah! Cool, i’ll start watching my switchports more closely then 🙂 Though i dont think it will effect us too much… Anyway, i was thinking: a daemon of some sort that extends SNMP, and returns the highest recorded burst over X milliseconds in the last Y minutes..?

        You could tune it for your monitoring system, and it should be easy(ish) to write the daemon.. But as usual i’m sure i’m out of my depth now…

        Cheers

  • Of course, take this to extreme – at any point in time an interface is either 100% busy (transmitting) or 100% idle (not transmitting.) (Just talking about one direction of the full duplex channel.)

    So hitting 1Mbit per millisecond just means you were running at line rate for about 90 packets in a row. (Assuming 1500 byte packets.) i.e. you sent 90 packets without much pause in between. But get to further granularity, and you’ll hit the 1Kbit per microsecond “limit” by sending a single packet.

    Doesn’t mean anything is wrong.

    The fact you are getting discards just means you are exceeding the switches buffer capacity for that port.  But the only way to know if the discards will affect performance is to measure them as a percentage of packets over a longer period (at Logicmonitor.com, we use 1 minute sampling for interfaces, not 5 minute. 5 minutes misses too many short lived peaks.) But seeing discards over a short period – even a “long” short period like a second – doesn’t matter. TCP is designed for discards. Systems like RED and WRED (weighted random early discard) increase line utilization by discarding (in some cases.) Discards that are “pretty frequent” may not matter, either – dropping 10 packets per second over a minute is not great – but if you are sending 1 million packets per second, its not a problem.

    So given you are exceeding the switches buffering for a port short term, IF you decide its a problem (which should be determined by the percentage of discards over a longer period) you can either add more buffering (by adding bonding, as you did) or tune the QoS on the switches. You can easily add more output buffers to the queue on ciscos.

  • Pingback: An interview with Kyle Brandt of Server Fault « Thuktun (Message)()

  • Pingback: SE Podcast #17 - Blog – Stack Exchange()

  • We just ran into the same thing on a cluster that was under heavy load.   After researching it, we found out it is a known issue called the “incast problem”.   One of our engineers blogged about it here: 

    http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-is-it/

  • Aries

    Hi Kyle,

    I see your point here. That makes wonder why the server is sending 900 Mbit per half second and 100Mbit per half second. how come it is not sending 1Mbit per millisecond?

    can you advise some ideas?

    thank you,