Broadcom, Die Mutha
Kyle Brandt
Until a year ago I never really thought much about NIC vendors. I figured if you bought Intel or Broadcom you were safe, maybe if I was doing near Gigabit speeds constantly or iSCSI I would have to pay more attention, but other than that I figured I was good.
Man, was I ever wrong.
Fail #1
This all started almost a year ago in our Oregon data center. In fact it was one the first things that was handed back to me when I started. On our web servers after upgrading to Windows Server 2008 R2 after about a week or two we would lose connectivity on a web server. This was happening to all the web servers, just not at the same time. After losing connectivity when you went to reboot the server it wouldn’t. You had to wait about 10 minutes for the BSOD to come and then the server would reboot (More at Windows Server 2008 R2 network adapter stops working, requires hard reboot).
So we raised a case with Microsoft and after a month of back and forth and some kernel patches we still had the problem. So we tried some Intel NICs and the problem went away.
Fail #2:
Now in our NY data center (Dell hardware instead of IBM, but still Broadcom NICs) I saw some packets being lost and various network timeouts recently. So I updated the firmware on a test server and a couple days later updated the rest of our web servers. I saw no improvements so I started to dig deeper into some tcpdump data with Wireshark and I see the following sorts of ARP requests coming from the servers:
17:03:41.187682 ARP, Request who-has 89.145.83.164 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 17:03:41.187684 ARP, Request who-has 74.125.91.109 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46 17:03:41.187686 ARP, Request who-has 64.34.80.179 (00:21:9b:a2:c9:be) tell 64.34.119.21, length 46ARP is used to find the MAC address of servers within the same network. In fact it is at the foundation of the network stack that much of the Internet is built on. There should never be ARP requests for IPs outside of your network (More at Windows 2008 R2 Servers Sending Arp Requests for IPs outside Subnet). This disappeared when I disabled failover teaming and came back when I enabled it again.
Fail #3:
Now about every week I am getting corrupted arp tables. Deleting the table with arp -d usually results in the table not being rebuilt, or if it does it shortly fails again after. The solution is to reboot the server. So I called Dell and they can’t help me because it is a software issue. So now we are right back where we were with our NICs in Oregon.
Solution:
We are replacing our Broadcom NICs with Intel on our primary production boxes. We replaced one of the NICs with an Intel NIC a couple weeks ago and have not seen either of these problems in that server so we are going to do this with the rest of our servers.
I don’t ever want to touch a Broadcom NIC again. Intel is a company that makes more sense to me anyways as their engineers are more frequently part of what I think of as the fabric of the Internet. Last issue I had with an Intel product I posted on their free mailing list and got a response from an Intel engineer within hours. The best thing to do at this point I think is to take these Broadcoms out to the field:
Pingback: Stack Overflow架构升级之路()
Pingback: Link-urile săptămânii 1-6 Martie | Staicu Ionuţ-Bogdan()
Pingback: [repost] Stack Overflow Architecture Update – Now at 95 Million Page Views a Month » New IT Farmer()
Pingback: Views of the Same Problem: Network Admin, DBA, and Developer - Server Fault Blog()
Pingback: Performance tuning Intel NICs - Server Fault Blog()
Pingback: Windows Server 2012 – A New Hope or Just New Hype? - Server Fault Blog()