HSRP or Hot Standby Router Protocol is a protocol designed to provide a virtual gateway for the purposes of fail over. The basic idea is that you have the gateway IP address shared between two or more devices and only one of those devices holds it at a time. If one of those devices fails another device will take hold of this IP address which is commonly referred to as the virtual IP (VIP). Other implementations of this are CARP and VRRP but for this discussion I will just use HSRP.
This solution is great to provide a redundant gateway for hosts on the LAN side of your network which is what it was designed for. RFC 2281 defines HSRP and is pretty clear about what it was designed for:
“Using HSRP, a set of routers work in concert to present the illusion of a single virtual router to the hosts on the LAN.”
“In particular, the protocol protects against the failure of the first hop router when the source host cannot learn the IP address of the first hop router dynamically. … HSRP is not intended as a replacement for existing dynamic router discovery mechanisms and those protocols should be used instead whenever possible.”
Despite this, I have had a couple of occasions where providers have tried to convince me that HSRP will provide the same level of redundancy as a dynamic routing protocol would for routers — in other words beyond the first hop. This setup turns out to be problematic.
The HSRP Setup and the Unemployed Network Administrator:
The setup for full network redundancy that the provider gives to the client’s network administrator has three different instances of HSRP. The first is a virtual gateway for the client from the provider (192.168.0.1). In addition to this the provider has the client run HSRP on the client’s WAN side so the provider can route 172.16.0.0/24 via the VIP 192.168.0.5. The third instance is for the client LAN (172.16.0.1), which is how HSRP is actually supposed to be used.
At first glance this looks fine. The network administrator goes through the diagram and asks himself for every router “What happens if this router fails?”. If Provider 1 fails then Provider 2 picks up the 192.168.0.1 VIP and the layer 2 switches make sure that the packets go to the right ports. The opposite is true if Provider 1 fails. The same basic scenario happens if one of the client routers fails as well. So the network administrator thinks, “Great, it works, I just need to explain this to my boss and then I can go home….” He goes to explain this to his annoying pointy haired boss and after the boss asks “What about the blue boxes?”. The network administrators says, “Oh you must mean the switches”, however he thinks to himself “You are not smart enough to even know what they are called — why do I have to explain this?”
So he hypothetically takes out Switch 1 and explains what happens as fast as he can in hopes to confuse his boss enough that he just gets to home. “In this scenario Provider 2 picks up the VIP of 192.168.0.1 and Client 2 picks up the VIP of 192.168.0.5. The result is that there is still a path to the web server.” After this Mr. Pointy Hair says, “Okay that looks right to me but I do want you to go test it.” The network administrator grumbles and agrees to test it even though he knows it is going to work just fine because not only does he know it will work but the provider also agrees and they do this for a living.
So the network administrator sets it all up and pulls the plug on the switch only to discover that it just doesn’t work — why?! The reason is that although Client 2 changed the WAN VIP, the LAN VIP still remains on Client 1. Therefore replies go to Client 1 and then have no way to reach the gateway on Provider 2. The network administrator for the client calls the provider and starts complaining that he is going to look like an incompetent to his boss now. The provider says they will look into it and then calls back the next day and says, “Don’t worry, you can have your LAN VIP track the WAN VIP so they stay synced up — If the WAN goes down the LAN will follow”. The network administrator does this in test and shuts down Switch 1 and to his great relief everything works. They deploy this in production and live happily ever after…
Until one day switch 3 fails. When this happens the mirror problem of switch 1 failing happens. The web server can send packets out via Client 2 but replies end up stuck on Client 1 because although the LAN tracks the WAN, the WAN doesn’t track the LAN. This problem is found by the network consultant that Mr. Pointy Hair hired to find out what the hell happened. Soon after this Mr. Network Administrator gets canned.
So what is the Solution? You might say the solution is to have both interfaces on the client routers track each other. Personally I never tried this and it might even work. However at this point if you get this to work you really missed the moral of the story. The purpose of HSRP is clearly documented and it tells you to use a dynamic routing protocol such as BGP, OPSF, ISIS, or EIGRP. The people who created this are probably either smarter than you are or spent more time thinking about it. You don’t want to hack it for a WAN by adding extra layer 2 devices and various configuration tricks. Do what the smart people say, if your provider disagrees don’t stop until they give you a proper solution. They can do it no matter what they say, you just have to convince them.