For a sysadmin, there is nothing quite like the bliss of being able to get to your stuff no matter what. Now we can go to extremes with terminal emulators hooked up to serial concentrators, that are then hooked into each server as well as having a modem so we can dial in. Personally I think that is a bit over kill unless you are running systems that people’s lives depend on.

Just like a good piece of cake, a proper Design-In-Depth remote access setup should be made of a few well selected complimentary system (or flavors depending on what you are currently working on!)

There is Cake

You should always be able to reach your servers somehow, even in the most extreme of circumstances. There are many technologies out there that allow you to manage servers remotely even when you can’t use the built in tools provided by the OS (ssh, RDP, etc).

There are three pieces to building a data center that you can (almost) always get to.

  • Out Of Band Management (OOB)
  • IP KVM
  • Network Accessible PDU’s

The most well known of these technologies right now is the OOB systems – most notable Dell’s DRAC and HP’s iLO system. These systems give you alot of control of the servers while they are not running an OS, but they fall short in a few areas, as well as can become unaccessable in certain situations.

IP KVM’s are a complementary system to the OOB systems. They are appliances just like normal KVMs that sit in your rack and allow you to access multiple machines with one keyboard/monitor/mouse. The big difference is that they have a network port, as well as remote access clients so that you don’t need to be physically in front of the Keyboard/monitor/mouse that is hooked up to it to use it. IP KVMs fill in the gaps as well as give you a redundant path to your server console. As we all know redundancy is a Good Thing.

I see you out there in the back of the auditorium … yes you waving you hand. You’re going to say

>”But, they do the same thing! Why do I need to pay for both? I get OOB with my server and those IP KVMs are pretty expensive.”

Well sir I have two answers for you. Firstly, this is part of design in depth, where you use complimentary technologies that run on different platforms to give you the most up time possible, as well as alternate paths to your equipment. Secondly, let me present a couple of scenarios to you.

  1. You have remote hands unpack and rack your brand new server. They hook everything up, but they cannot figure out your particular brand of OOB card. How do you go in and configure it yourself easily? Or how do you avoid spending 40 minutes on the phone walking them through setting up the card?
  2. It’s firmware update time of year! Yea! So you patch the OOB card’s firmware. when you reboot the server it doesn’t come back up and you can’t connect to the OOB card! How would you view the console? Oh and the server is 500 miles away in a Colo that only provides hands from 9-5 M-F and it’s now 3am on Saturday because you did this during a low traffic maintenance window. How would you get to the machine to see that the firmware update reset the IP on the OOB card and boot was stalled because the OOB card hung waiting for you to reset it?

Your final line of defense is going to be network accessible PDU’s while they have many great qualities, the one that applies most to a remote access discussion is the ability to remotely power cycle the ports. If you every need to kick over a server that is half a continent away you will grow to appreciate these. Just make sure that you have your servers set to “start on power failure” in the BIOS.

What is the moral of this story? Well it’s a concept I like to call design-in-depth basically the sysadmin equivalent to the security people’s concept of security-in-depth. You build a system that using different technologies that are complimentary to each other, and provide some overlap in functionality. This will allow you to someway, somehow fix what is broken right now. It also gives you options when you run into an issue with a particular technology. The more tools you put into your sysadmin toolbox, the better chance you have of achieving the goal of maximum up time with minimal user inconvenience.

I am excited to announce that Stack Overflow is doubling the size the of the sysadmin team. We have picked up another member of the community and that member is George Beech. The regular users of Server Fault probably know him as Zypher:

Server Fault profile for Zypher

From looking at George’s answers on Server Fault and chatting with him in the Server Fault chat rooms I can tell you that George is a very knowledgeable sysadmin and also a cool guy that I am pumped to start working with. On Server Fault I have seen questions about server to admin ratios. I tend to agree that you can not prescribe a number. However, for a company with a popular web site I say at least two. I think all sysadmins should have a sysadmining buddy. Also between trying to exemplary sysadmins, writing blog posts, supporting employees, and dealing with the problems such as attacks on your website being a solo sysadmin is not ideal. Having two sysadmins is probably the most important place to have redundancy in your infrastructure. Lastly, a second person as great as George is going to provide all sorts of ideas and experience beyond what I could come up with.

So congratulations George, I look forward to our adventure!

When I started to build the the new data center I went with the Active Directory name One advantage with this split DNS setup is that you can have internal IPs for certain entries. I knew I would have to duplicate domain entries but I did not mind that and I liked the aesthetic so I made that call.

In most cases I don’t think it is really that bad but we happen to run a popular web site that uses that same name. If the developers have to make URL based calls to the site from the site than it would have to be worked around. Other downsides include that the duplicate entries could be wrong. The vote on Server Fault also seems to be against it. So on reflection, this was simply the wrong call.

So yesterday I went about changing the AD domain name to a sub domain. I also was able to keep STACKOVERFLOW as the short name for the aesthetic. This has been a pain and I think there are still probably some things to fix but in the long run it will pay off.

The moral of the story is that when you make a bad decision then suck it up and fix it. I think this is especially true at the core of a system. If something is wrong there than you have to work around that, and then you have to work around your workaround. What you end up with is:

workaround = "workaround"
while workaround:
    workaround += " the " + workaround

So, if you can, just fix your bad decisions.

I have spent the past couple of months getting our new data center in New York ready to start moving our production applications to. As I have built this out there have been two areas of administration which I have paid particular attention to:

  • Scalability of Management
  • Fault Tolerance

Scalability of management means to me that I am ahead of the curve when it comes to my method of administration relative to the size of the infrastructure. So if there are 20 servers I want to administer them as if there were 100. The infrastructure right now is:

  • 10 Windows IIS Web Servers
  • 2 Windows MS SQL Database Servers
  • 2 Linux Routers
  • 2 Linux Load Balancers with HAProxy
  • 1 Linux Backup Server running Bacula
  • 1 Linux Management Server for Nagios and Logs
  • 2 VMWare ESXi Servers
  • 5 Dell Power Connect Switches

It has been a lot of work getting all these operating systems installed, servers wired up (see the image below of one of two cabinets), and everything configured.

However, I have made it into a lot more work that it might have been. Why? It wasn’t because I labeled each end of each cable as much as it is that I have aimed to build out everything in a way that the management of it will scale. More often than not it would have been faster to go and do everything manually on each server. However I have considered that bad for a few main reasons:

If I gain the experience and knowledge of how to manage these servers as if there were at least 3 times as many I won’t have to deal with certain management issues when the pressure is on. I am sure that sort of growth will present all sort of other problems that I will be scrambling to solve. The analogy Jeff Atwood has used for the community problems we have faced as we have grown is big city vs. small city planning. I want to be as ready as possible to face the big city problems. I also believe in the long run this philosophy saves time. Lastly, manually administering each server as a single entity is just lame and pedestrian.

Some of the tools I have used that I believe will aid in scaling my management are as follows:

A Windows Deployment Server has allowed me to set up fully unattended installs of Windows for my servers. I was able to have 10 windows installs with all of them joined to the windows domain in just one morning. Of course it took me about a day to get the unattended configuration working after testing installs on VMs but in the long run I think it is worth it. Deploying servers as we grow will be that much easier.

Group Policies for Windows has been essential to this aspect. These have allowed me to get much of the needed software installed either by using the msi installation feature or by setting up boot up scripts with them. This has included things such as enabling remote desktop, installing software packages such as NSClient++ and Mercurial, configuring firewall rules and deploying roles such as SNMP, IIS and .NET. The software installation aspect really can’t compare to something like apt for Debian but nonetheless it is essential to take advantage of what I can.

Kludgy power shell scripts have been useful to for copying configuration files to all the servers and allowing me to restart the service. Although I have done a lot of bash shell scripting, power shell is new to me. However I was able to copy the NSClient ini file and restart the service using just a little snippet I picked up in not too long:

$webservers = @("ny-web01", "ny-web02", "ny-web03", "ny-web04", "ny-web05", "ny-web06", "ny-web07", "ny-web08", "ny-web09")

foreach ($server in $webservers) { Write-Host $server cp 'C:Program FilesNSClient++NSC.ini' "\$serverc$Program FilesNSCLient++NSC.ini" $svc = Get-WmiObject Win32_Service -ComputerName $server -Filter "name='nsclientpp'" if ($svc.started -eq $true) { $svc.StopService() } $svc.StartService() }

This as well had a higher up front time cost than just manually copying the file, but in the long run it is better administration, it scales, and saves time. I have already had to make some changes to that configuration and pushing the changes now is fast and simple.

I have deployed a Windows Update Server (WSUS) to allow me to control the updates that go to certain servers from a centralized location. I opted to have the updates are scheduled via a group policy and controlled with WSUS by using client-side targeting.

Nagios has also been essential in this philosophy in that it uses an inheritance and template system. Because of this once all the services and groups are configured all I have to do is add the host the group to getting everything monitored. To further extended this I have used n2rrd for graphing which plugs directly into Nagios. This way adding checks and graphing to a new server deployment is all done in a single step. Also if I want to add a new check to all the servers once I get it setup I just have it run for the entire group which is as simple as adding it the host group. Gathering the data in the graph means storing the results over time which is essential for trending so that new hardware can be purchased ahead of time.

All these tools are the tools I think that are needed to build a city. Some tools are good for a server or two but they just don’t scale.

There is still more to do. For the Linux boxes I have been using ssh and loop scripts similar to the power shell script above and really something like Puppet is more appropriate. Also not every piece of software had a simple way to deploy via a group policy. Sadly enough, .NET 4 doesn’t come as an MSI and isn’t available in the update services yet (but is coming “soon”, as in 2-12 months). Also a centralized logging solution is on my to do list as checking the logs on each server is quite tedious.

Fault Tolerance: Fault tolerance is basically the idea that if one component fails it doesn’t bring everything down. I am of the belief the network is a good area to start with fault tolerance and this is where I have put much of my efforts.

Network Fault Tolerance: In a previous Server Fault Blog post entitled “HSRP is not for WANs” I explained why I didn’t think HSRP was valid for the WAN and that a routing protocol should be used. I managed to convince our collocation provider Peer 1 of this and we now have a private BGP peering with them. On our 2 Linux routers I use the Quagga routing suite for BGP. ucarp is used to provide a redundant gateway for the LAN. For stateful firewall rules in IPTables I use conntrackd to make sure that any asymmetric routing patterns don’t break because of mismatched state tables.

Each server is connected to two different switches and has bonding in an active fail over mode.

With all of this going on the key has been to test the fail over by bring down the routers and interfaces on the routers. Without the tests you are only hoping you have fault tolerance.

Power Fault Tolerance in Thirds: Power can be tricky for fail over. Many people think you just put two power supplies in each server connected to independent feeds and you are good to go. However, I had learned from others experiences that at least with the older Dell PowerEdge servers you didn’t know which power supply would be active. If you don’t properly watch your power when one of the feeds fails you can overload the other and have no power. So the alternative I have taken is to divide the servers into groups of 3 (when there are more than two). One server will have dual power supplies, the second will be connected to the A feed and the third server to the B feed. This way if power fails we will be operating at 66% instead of 50% capacity. In general when there are only 2 servers the second is a warm backup so I just make sure they are on different feeds. If there is only one server that I have dual power supplies in it).

Fault Prevention: I have also made sure to set up a staging server in the production environment. The best way to handle faults is to stop them from happening and I think having a test server should cut down on them quite a bit.

I think scalability of management and fault tolerance are just two aspects that make the difference between administering servers and being a professional system administrator. These are the sort of topics that make system administration a profession and it is one reason why Server Fault can be such a great resource for system administrators.

Server Fault now has a place for real time chat over at We have had a room going for a few weeks now over at but it is nice for us to have our own place now where we can create specific chat rooms now.

A place for live chat is a great thing to have for system administrators. A lot of system administrators seem to be out working on their own so this is a nice place to socialize. It also can be a place to talk about those questions that don’t really have straightforward answers. For example:

If you haven’t seen the chat system that Stack Overflow’s employees Benjamin Dumke and Marc Gravell have developed you are missing out. I bet a lot of system administrators, especially grumpy ones with a *nix slant like me, might have a first thought similar to what I had:

Why do we need some Web 2.0 garbage to replace our wonderful text based IRC?

It turns out there a lot of reasons besides all the damn netsplits. This chat retains all the history so when you join a room you can see what was being talked about, you can also search the transcript. There is a lot more as well — I recommend you go check it out.