For a sysadmin, there is nothing quite like the bliss of being able to get to your stuff no matter what. Now we can go to extremes with terminal emulators hooked up to serial concentrators, that are then hooked into each server as well as having a modem so we can dial in. Personally I think that is a bit over kill unless you are running systems that people’s lives depend on.

Just like a good piece of cake, a proper Design-In-Depth remote access setup should be made of a few well selected complimentary system (or flavors depending on what you are currently working on!)

You should always be able to reach your servers somehow, even in the most extreme of circumstances. There are many technologies out there that allow you to manage servers remotely even when you can’t use the built in tools provided by the OS (ssh, RDP, etc).

There are three pieces to building a data center that you can (almost) always get to.

• Out Of Band Management (OOB)
• IP KVM
• Network Accessible PDU’s

The most well known of these technologies right now is the OOB systems – most notable Dell’s DRAC and HP’s iLO system. These systems give you alot of control of the servers while they are not running an OS, but they fall short in a few areas, as well as can become unaccessable in certain situations.

IP KVM’s are a complementary system to the OOB systems. They are appliances just like normal KVMs that sit in your rack and allow you to access multiple machines with one keyboard/monitor/mouse. The big difference is that they have a network port, as well as remote access clients so that you don’t need to be physically in front of the Keyboard/monitor/mouse that is hooked up to it to use it. IP KVMs fill in the gaps as well as give you a redundant path to your server console. As we all know redundancy is a Good Thing.

I see you out there in the back of the auditorium … yes you waving you hand. You’re going to say

>”But, they do the same thing! Why do I need to pay for both? I get OOB with my server and those IP KVMs are pretty expensive.”

Well sir I have two answers for you. Firstly, this is part of design in depth, where you use complimentary technologies that run on different platforms to give you the most up time possible, as well as alternate paths to your equipment. Secondly, let me present a couple of scenarios to you.

1. You have remote hands unpack and rack your brand new server. They hook everything up, but they cannot figure out your particular brand of OOB card. How do you go in and configure it yourself easily? Or how do you avoid spending 40 minutes on the phone walking them through setting up the card?
2. It’s firmware update time of year! Yea! So you patch the OOB card’s firmware. when you reboot the server it doesn’t come back up and you can’t connect to the OOB card! How would you view the console? Oh and the server is 500 miles away in a Colo that only provides hands from 9-5 M-F and it’s now 3am on Saturday because you did this during a low traffic maintenance window. How would you get to the machine to see that the firmware update reset the IP on the OOB card and boot was stalled because the OOB card hung waiting for you to reset it?

Your final line of defense is going to be network accessible PDU’s while they have many great qualities, the one that applies most to a remote access discussion is the ability to remotely power cycle the ports. If you every need to kick over a server that is half a continent away you will grow to appreciate these. Just make sure that you have your servers set to “start on power failure” in the BIOS.

What is the moral of this story? Well it’s a concept I like to call design-in-depth basically the sysadmin equivalent to the security people’s concept of security-in-depth. You build a system that using different technologies that are complimentary to each other, and provide some overlap in functionality. This will allow you to someway, somehow fix what is broken right now. It also gives you options when you run into an issue with a particular technology. The more tools you put into your sysadmin toolbox, the better chance you have of achieving the goal of maximum up time with minimal user inconvenience.