Broadcom, Die Mutha

Kyle Brandt

Until a year ago I never really thought much about NIC vendors. I figured if you bought Intel or Broadcom you were safe, maybe if I was doing near Gigabit speeds constantly or iSCSI I would have to pay more attention, but other than that I figured I was good.

Man, was I ever wrong.

Fail #1

This all started almost a year ago in our Oregon data center. In fact it was one the first things that was handed back to me when I started. On our web servers after upgrading to Windows Server 2008 R2 after about a week or two we would lose connectivity on a web server. This was happening to all the web servers, just not at the same time. After losing connectivity when you went to reboot the server it wouldn’t. You had to wait about 10 minutes for the BSOD to come and then the server would reboot (More at Windows Server 2008 R2 network adapter stops working, requires hard reboot).

So we raised a case with Microsoft and after a month of back and forth and some kernel patches we still had the problem. So we tried some Intel NICs and the problem went away.

Fail #2:

Now in our NY data center (Dell hardware instead of IBM, but still Broadcom NICs) I saw some packets being lost and various network timeouts recently. So I updated the firmware on a test server and a couple days later updated the rest of our web servers. I saw no improvements so I started to dig deeper into some tcpdump data with Wireshark and I see the following sorts of ARP requests coming from the servers:

17:03:41.187682 ARP, Request who-has 89.145.83.164 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 
17:03:41.187684 ARP, Request who-has 74.125.91.109 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 
17:03:41.187686 ARP, Request who-has 64.34.80.179 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46
ARP is used to find the MAC address of servers within the same network. In fact it is at the foundation of the network stack that much of the Internet is built on. There should never be ARP requests for IPs outside of your network (More at Windows 2008 R2 Servers Sending Arp Requests for IPs outside Subnet). This disappeared when I disabled failover teaming and came back when I enabled it again.

Fail #3:

Now about every week I am getting corrupted arp tables. Deleting the table with arp -d usually results in the table not being rebuilt, or if it does it shortly fails again after. The solution is to reboot the server. So I called Dell and they can’t help me because it is a software issue. So now we are right back where we were with our NICs in Oregon.

Solution:

We are replacing our Broadcom NICs with Intel on our primary production boxes. We replaced one of the NICs with an Intel NIC a couple weeks ago and have not seen either of these problems in that server so we are going to do this with the rest of our servers.

I don’t ever want to touch a Broadcom NIC again. Intel is a company that makes more sense to me anyways as their engineers are more frequently part of what I think of as the fabric of the Internet. Last issue I had with an Intel product I posted on their free mailing list and got a response from an Intel engineer within hours. The best thing to do at this point I think is to take these Broadcoms out to the field:

«
»
  • http://twitter.com/rusanu Remus Rusanu

    Damn It Feels Good to Be a Gangsta…

    Couldn’t help it when I saw the picture

  • http://stoneyforest.net/~chris/ Chris S

    I find it a bit humorous that we use nothing but Broadcom NICs, actually took a few Intel NICs out of service to standardize. I’ve never had a single problem with any of them, but we also only use HP ProLiant Servers, perhaps this has an effect. I don’t have access to any Dell or IBM servers, but it would make an interesting experiment.

    I have heard some horror stories about using cheap NICs however. ServerFault has quite a few examples of people using Realtec or VIA NICs with the full spectrum of odd behavior and poor performance.

    • http://codinghorror.myopenid.com/ Jeff Atwood

      it’s very specific to Windows Server 2008 R2 which has the Windows 7 kernel / stack. We didn’t see this in Windows Server 2008 (which has the Vista kernel / stack).

      • http://stoneyforest.net/~chris/ Chris S

        We currently run Hyper-V Server 2008 R2 or Windows Server 2008 R2 DC on our servers, still no problems. It’s possible that the new stack takes advantage of hardware in new and different ways, where certain combinations break. This should not happen, obviously; but it is one potential explanation.

      • http://sysadmin1138.net/ sysadmin1138

        At the moment all of our R2 is in ESX, so we’re shielded from this. However, I’ve had enough Broadcom problems on Windows 2003 that it’s something I think of whenever I see weird crap on the wire like in fail #2.

    • VSeven

       I have three 2008 R2 HyperV hosts all with Broadcom cards (dual and quad nics).  The only way I could get decent performance was to disable TCP offload and chimney.  My backups from server to server (1Gb Broadcom on each throguh to 1Gb switch) went from 13 hours to 8 hours after disabling.  The latest Boardcom driver update reenabled all the settings and would NOT let me disable it on the host machine (but did let me disable it on each of the VM nics).  So my backup went from 8h back up to 13.  Rolled back the driver to a old one and back down to 8h.  Next set of server nics will be Intel unless Boardcom can fix there issues in a hurry.

       

    • Prmarino1

      Ive seen it on HP equipment. I dont have the data handy but a few years ago  the financial company i worked for did some real intensive testing on intel and braodcom nics because our standard builds all included Intel nics. the goal was to see if we could save some money on our up coming order of 1000 HP Proliant servers by using the integrated broadcom NIC’s.

      the results were surprising. TCP trafic was 20% slower on broadcoms. UDP drop rate when pushing the nic to near capacity on a 1GB network was astonishing on the broadcom nics.

      My advice is broadcom is fine for light use or as an out of band management interface but unacceptable for UDP streaming or any environment that has real time or virtualization requirements.  Simply its a desktop NIC that gets stuck in servers because they are cheap and claim to support many features that sound like things that you would want in an enterprise environment, but when you use them you get what you pay for.

      If a company is using it broadcom NIC’s on a server that has a high transaction rate and you isn’t seeing issues its because either they aren’t really looking for them or more commonly people see the problems and think that issues they’ve been seeing are normal so they don’t recognize them for what they are.

  • http://twitter.com/Koesper Casper de Groot
  • http://www.facebook.com/dpavlin Dobrica Pavlinušić

    I think that your hate towards Broadcom is somewhat based on your expirience of Windows and Broadcom. I never had similar problems using Broadcom with Linux, so YMMV Broadcom users…

    • http://kyle-brandt.myopenid.com/ Kyle Brandt

      Yes these problems have been on our Windows 2008 R2 Servers and only our Web Servers in both OR and NY. However, in NY this seems to have started happening after the firmware upgrade and I only did that on the NY web servers. Although if you have been a long time Desktop Linux user you might have a couple bad years of memories with Broadcom’s Wireless G NICs and ndis wrapper.

    • http://codinghorror.myopenid.com/ Jeff Atwood

      yes, it is definitely advice intended mostly for Windows sysadmins at the moment.

    • http://kyle-brandt.myopenid.com/ Kyle Brandt

      Also, being specific to Windows 2008 R2 is like a website only having issues with the current Internet Explorer.

  • http://holocryptic.myopenid.com/ Holocryptic

    I die a little inside every time I see a Broadcom installed in a server… which is a lot. By all rights, I should be dead.

  • http://holocryptic.myopenid.com/ Holocryptic

    I die a little inside every time I see a Broadcom installed in a server… which is a lot. By all rights, I should be dead.

  • http://openid.palahala.nl/ Arjan

    I am not sure if it’s related to NICs, nor what NIC my Dell computer at a previous job was using. Still, just for Google’s sake: some Dell workstations at work would reboot when publishing Magnolia CMS content using HTTP over localhost. The event logs sometimes referred to bad network card drivers.

  • http://pulse.yahoo.com/_UJPRT2S637VVGD2VJQSFPMMHGY Jeroen Pluimers

    On Windows, I’ve had my share of issues with Marvell NICs too: they tend to loose their connection without telling me; solution is much simpler than with Broadcom: just disable/enable the Marvell NIC and it comes up fine, then works for a random amount of time.

    This is on varios x86 Windows 2003 systems though.

    I never had issues with Broadcom under VMware ESXi. So it might be NIC/driver related.

    Public systems are pure Intel, no bad words about them: more expensive than a lot of the competition, but really good quality.

  • http://kevinkenny.myopenid.com/ Kev

    We’ve got a bunch of Dell R710′s with Broadcom BCM5709C NetXtreme II GigE NIC’s running Windows 2008R2 (RTM not SP1) which are under constant heavy load 24/7 and we’ve never had a problem. That said, your problem noted just in case we do see any oddness.

  • Pingback: Stack Overflow架构升级之路

  • Pingback: Link-urile săptămânii 1-6 Martie | Staicu Ionuţ-Bogdan

  • Pingback: [repost] Stack Overflow Architecture Update – Now at 95 Million Page Views a Month » New IT Farmer

  • Pingback: Views of the Same Problem: Network Admin, DBA, and Developer - Server Fault Blog

  • Pingback: Performance tuning Intel NICs - Server Fault Blog

  • haus

    We started seeing issues with systems with this type of broadcom NIC:

    (from linux dmsg): eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express

    On the network side, we saw the switches complaining of errors related to flow control. Basically the switches were receiving pause frames from the server nic in such an amount that the switch eventually starts discarding frames… certain switches have this feature enabled by default (cisco 4958-e and likely others on the 4500 platform). The 6500 platform has flow control disabled by default, so the problem never occurred for us on the past.

  • http://openid-provider.appspot.com/james.yale James Yale

    Just found this linked to from a ServerFault post – very interesting as I’ve been fighting a problem where our bonded broadcom NICs on our main file server (Dell PE2960) will seemingly randomly stop transmitting on RHEL5.5 – don’t have the problem on Sun box with the same software config but nvidia NICs.

    When it happens the bond running with the broadcom NICs always stops working, while another bond on the box running with a pair of Intel adapters keep going – usually the only solution is to reboot the box.

    Obviously I’ve got nothing to prove this is the same issue at this point, but the symptoms are remarkably similar.

    • Sean Keeney

      I know this post was a while ago, but you might have the issue we had in our DC – the NICs simply stopped working but the driver didn’t indicate anything wrong, and transmission restarted after pulling the cable or restarting the networking service.

      The ‘fix’ is to disable msi-x in modprobe.conf for the driver.

      Broadcom drivers are woeful…

  • http://www.ppetrov.net/ pip010

    I recommend 3Com. Their superb drivers and hardware tcp/ip stack acceleration is a must for any GRID system. Broadcom are good enough for consumers, cheep 1Gbit net. Although in this case is the NEW EFATURE (a.k.a bug) http://support.dell.com/support/edocs/network/p29352/english/teaming.htm

  • http://profiles.google.com/wim.kerkhoff Wim Kerkhoff

    These Broadcom issues are not specific to Windows. Even on Linux with 2.6.38 the drivers are less then ideal. Weird latencies, dropped packets, etc…

  • Guixingyi

    several vendors have server hardware shipment with Broadcom nics which are integrated within motherboard, like IBM X series server, we don’t have any choice if the issue happen, except to purchase new additional pci nics.

  • http://stoneyforest.net/~chris/ Chris S

    Quick update: Over time I’ve noted that everyone who has problems with Broadcom NICs are putting them in Dell or IBM servers. Not a single problem report from HP gear. I believe this factors into the problems somehow, but still can’t fully explain it. Regardless my recommendation at this time is not to use Broadcom NICs in Dell or IBM servers.

  • Pingback: Windows Server 2012 – A New Hope or Just New Hype? - Server Fault Blog