I have spent the past couple of months getting our new data center in New York ready to start moving our production applications to. As I have built this out there have been two areas of administration which I have paid particular attention to:

  • Scalability of Management
  • Fault Tolerance

Scalability of management means to me that I am ahead of the curve when it comes to my method of administration relative to the size of the infrastructure. So if there are 20 servers I want to administer them as if there were 100. The infrastructure right now is:

  • 10 Windows IIS Web Servers
  • 2 Windows MS SQL Database Servers
  • 2 Linux Routers
  • 2 Linux Load Balancers with HAProxy
  • 1 Linux Backup Server running Bacula
  • 1 Linux Management Server for Nagios and Logs
  • 2 VMWare ESXi Servers
  • 5 Dell Power Connect Switches

It has been a lot of work getting all these operating systems installed, servers wired up (see the image below of one of two cabinets), and everything configured.

However, I have made it into a lot more work that it might have been. Why? It wasn’t because I labeled each end of each cable as much as it is that I have aimed to build out everything in a way that the management of it will scale. More often than not it would have been faster to go and do everything manually on each server. However I have considered that bad for a few main reasons:

If I gain the experience and knowledge of how to manage these servers as if there were at least 3 times as many I won’t have to deal with certain management issues when the pressure is on. I am sure that sort of growth will present all sort of other problems that I will be scrambling to solve. The analogy Jeff Atwood has used for the community problems we have faced as we have grown is big city vs. small city planning. I want to be as ready as possible to face the big city problems. I also believe in the long run this philosophy saves time. Lastly, manually administering each server as a single entity is just lame and pedestrian.

Some of the tools I have used that I believe will aid in scaling my management are as follows:

A Windows Deployment Server has allowed me to set up fully unattended installs of Windows for my servers. I was able to have 10 windows installs with all of them joined to the windows domain in just one morning. Of course it took me about a day to get the unattended configuration working after testing installs on VMs but in the long run I think it is worth it. Deploying servers as we grow will be that much easier.

Group Policies for Windows has been essential to this aspect. These have allowed me to get much of the needed software installed either by using the msi installation feature or by setting up boot up scripts with them. This has included things such as enabling remote desktop, installing software packages such as NSClient++ and Mercurial, configuring firewall rules and deploying roles such as SNMP, IIS and .NET. The software installation aspect really can’t compare to something like apt for Debian but nonetheless it is essential to take advantage of what I can.

Kludgy power shell scripts have been useful to for copying configuration files to all the servers and allowing me to restart the service. Although I have done a lot of bash shell scripting, power shell is new to me. However I was able to copy the NSClient ini file and restart the service using just a little snippet I picked up in not too long:

$webservers = @("ny-web01", "ny-web02", "ny-web03", "ny-web04", "ny-web05", "ny-web06", "ny-web07", "ny-web08", "ny-web09")

foreach ($server in $webservers) { Write-Host $server cp 'C:Program FilesNSClient++NSC.ini' "\$serverc$Program FilesNSCLient++NSC.ini" $svc = Get-WmiObject Win32_Service -ComputerName $server -Filter "name='nsclientpp'" if ($svc.started -eq $true) { $svc.StopService() } $svc.StartService() }

This as well had a higher up front time cost than just manually copying the file, but in the long run it is better administration, it scales, and saves time. I have already had to make some changes to that configuration and pushing the changes now is fast and simple.

I have deployed a Windows Update Server (WSUS) to allow me to control the updates that go to certain servers from a centralized location. I opted to have the updates are scheduled via a group policy and controlled with WSUS by using client-side targeting.

Nagios has also been essential in this philosophy in that it uses an inheritance and template system. Because of this once all the services and groups are configured all I have to do is add the host the group to getting everything monitored. To further extended this I have used n2rrd for graphing which plugs directly into Nagios. This way adding checks and graphing to a new server deployment is all done in a single step. Also if I want to add a new check to all the servers once I get it setup I just have it run for the entire group which is as simple as adding it the host group. Gathering the data in the graph means storing the results over time which is essential for trending so that new hardware can be purchased ahead of time.

All these tools are the tools I think that are needed to build a city. Some tools are good for a server or two but they just don’t scale.

There is still more to do. For the Linux boxes I have been using ssh and loop scripts similar to the power shell script above and really something like Puppet is more appropriate. Also not every piece of software had a simple way to deploy via a group policy. Sadly enough, .NET 4 doesn’t come as an MSI and isn’t available in the update services yet (but is coming “soon”, as in 2-12 months). Also a centralized logging solution is on my to do list as checking the logs on each server is quite tedious.

Fault Tolerance: Fault tolerance is basically the idea that if one component fails it doesn’t bring everything down. I am of the belief the network is a good area to start with fault tolerance and this is where I have put much of my efforts.

Network Fault Tolerance: In a previous Server Fault Blog post entitled “HSRP is not for WANs” I explained why I didn’t think HSRP was valid for the WAN and that a routing protocol should be used. I managed to convince our collocation provider Peer 1 of this and we now have a private BGP peering with them. On our 2 Linux routers I use the Quagga routing suite for BGP. ucarp is used to provide a redundant gateway for the LAN. For stateful firewall rules in IPTables I use conntrackd to make sure that any asymmetric routing patterns don’t break because of mismatched state tables.

Each server is connected to two different switches and has bonding in an active fail over mode.

With all of this going on the key has been to test the fail over by bring down the routers and interfaces on the routers. Without the tests you are only hoping you have fault tolerance.

Power Fault Tolerance in Thirds: Power can be tricky for fail over. Many people think you just put two power supplies in each server connected to independent feeds and you are good to go. However, I had learned from others experiences that at least with the older Dell PowerEdge servers you didn’t know which power supply would be active. If you don’t properly watch your power when one of the feeds fails you can overload the other and have no power. So the alternative I have taken is to divide the servers into groups of 3 (when there are more than two). One server will have dual power supplies, the second will be connected to the A feed and the third server to the B feed. This way if power fails we will be operating at 66% instead of 50% capacity. In general when there are only 2 servers the second is a warm backup so I just make sure they are on different feeds. If there is only one server that I have dual power supplies in it).

Fault Prevention: I have also made sure to set up a staging server in the production environment. The best way to handle faults is to stop them from happening and I think having a test server should cut down on them quite a bit.

I think scalability of management and fault tolerance are just two aspects that make the difference between administering servers and being a professional system administrator. These are the sort of topics that make system administration a profession and it is one reason why Server Fault can be such a great resource for system administrators.

«
»