We have been live at the New York data center with stackoverflow.com, superuser.com, serverfault.com and the rest of the Stack Exchange network for a work week now and the SysAdmin team is pleased to say that it has gone well. It has been smooth since launch with the only downtime being for some quick tweaking we wanted to try.

So how does the performance look when it comes to our improved latency promise?

We use Pingdom to monitor our servers from outside our network. This data is the average time to load the Stackoverflow.com homepage from all the Pingdom locations which includes locations throughout Europe, The United States, and Canada to load the Stackoverflow.com homepage. Given, this is not exactly scientific, but the big drop in starting the date of the move (October 24th) is a big win in my book. When I, well, the Pingdom web interface, ran these numbers for the Texas and Chicago I also saw an improvement in the latency.

Diagrams Please
When we got an email of someone offering to do a network diagram for us I knew it was time to get something up:

This is just an overview of the network — on the left is our public facing network and the right side is our private network. There might be some reworking of the network layer down the road (more on that later).

The Web Servers:
We now have 4 more web servers which are also more powerful. These are all Windows Server 2008 R2.

Old Servers:

New Servers:

We now keep all the web server configs the same so adding another server for a site is just a load balancer change. So currently stackoverflow.com runs on 6 servers (3 servers dedicated for stackoverflow only).

The CPU graphs on our web servers show just how much of better performers they are for us. The following is a comparison of dedicated stackoverflow.com web servers (Keep in mind that these are on 6 servers instead of 3 now).


New: In general the CPU load on our web servers has been balanced and it still is. In short, we have plenty of room for growth here.

The Database Servers:
Our database servers are also more powerful. Old Database Servers:

New Database Servers:

We are currently working through a full text performance regression (Possibly going from SQL 2008 to R2) and will upload graphs when we have a more meaningful comparison.

The Network Layer:
Our router/firewall are both Linux machines with the same basic specs as the web Tier but with less memory. My goal has been perfect redundancy at the network layer and to eliminate all SPOFs. To this end we have two switches, CARP on our LAN and a private BGP peering with our provider, redundant switches and bonded nics. I got close but tests show that I have failed. Right now at the heart of this problem there are two things:

  1. Quagga just doesn’t work the way I would expect dynamic routing protocols to work. If a route disappears it keeps announcing it unless I redistribute static routes. Also, if a connected route goes down and it also knows about the route through iBGP it doesn’t insert the iBGP route into the table but does remove the static route.
  2. Stateful Firewalls and asymmetric routes don’t like each other. The fundamental problem is that they would need to sync state faster than SYN and then the SYN ACK or suffer performance problems. There is an interesting whitepaper on this topic: “Demystifying cluster-based fault-tolerant Firewalls.”

George and I were left with two options, try to force symmetry in the entire network or eliminate statefulness. After a day of back and forth we have decided that Linux/Quagga isn’t working for us and that we also need to separate out our router and firewall levels. Our plan is currently to move to Cisco because we are familiar with them and the time cost to explore Juniper or BSD is not appealing to us at the moment. Currently however we can lose either the router our either WAN link and still automatically come back up. So do have redundancy, it just isn’t perfect yet.

Redis and our Load Balancers:
These are also currently spec’ed out the same as our web tier machines except that the load balancers have 4 GB or RAM. They run Linux (Currently a mix of Ubuntu Server and CentOS). If you have been following this blog you know we use HAProxy and love it. We also use Redis as our distributed caching layer.

Internal Network:
Besides the database our Internal network has the following:

  • A Linux backup Server running Bacula which also handles our web logs and web log analysis (R610, 32 GB Ram, Xeon E5640)
  • A Linux monitoring Server with Nagios and Splunk and n2rrd for Graphs. (R610, 32 GB Ram, Xeon 5640)
  • 2 ESX Servers for our Domain Controllers (R610, 16 GB Ram, Xeon 5640)

From a sysadmin perspective, all that ram gives us the ultimate log analysis playground real estate.

What is next?
We still have to move our sstatic.com domain. George is currently building out our in house external facing DNS servers. We also still have our Oregon data center so some geographic load balancing of the static content is on our wishlist. Also, we are trying to gather all the stats we can into Nagios so we can measure the impact of our tweaks, trend, solve problems quickly, and enjoy the info porn.

We scale in three directions up, out, and awesome people. We don’t have lots of small weak servers (out) nor do we have a couple of monsters (up) — rather we strike a balance. Our third, and most import scaling direction is getting awesome people. By that I mean we have great programmers who happen to be performance junkies. They literally tune the code 24/7 and also know a lot about SQL and system tuning as well. The sysadmin team will be working with them to get the best damn performance we can from our new machines.

We are starting the move of our services from our Oregon Data center to Our Brand-Spanking-Shiny-New NYC Data Center.

We’ll do our best to keep things up and running while we move but there may be some service interruption and oddities as we get everything over and running.

We should have all the migrations done sometime today (but sooner than 6-8 weeks)

Check back here for updates on the move.

Much ado about DNS

George Beech

DNS is one of those systems that is one of the easiest to setup, yet it is one of the hardest to setup right. Most of the time just works is good enough for probably 90% of the installations out there. But as system administrators, just works should never be what we strive for. We should always go for the best implementation that we possibly can achieve.

There are basically two choices you need to make when you want to setup your DNS.

  • Do I want to host DNS internally
  • Do I want to host DNS externally


Both of these options have their advantages and drawbacks. In broad strokes they can be listed as:


Hosted Solution’s

  • Low management overhead – you don’t need to worry about the hardware, high availability and distribution of server
  • Low learning curve – You are generally given a web interface that makes it very simple to add records, with explanations for what each of them do to help you along
  • Low cost to get in, on the low end of traffic levels very affordable.

In House

  • Highly customizable
  • You aren’t reliant on another entity for your DNS
  • When your traffic starts to climb, the cost/benefit ration is squarely in the favor of in house hosting


Hosted Solutions

  • At a certain volume of traffic they can become very expensive
  • Making many changes can be painful without writing code against their API, if they have one
  • You are reliant on another entity to keep a core service running

In House

  • You have to put out the hardware cost up front
  • You need to have people with the proper skills to manage the servers properly
  • You need to have the a large enough infrastructure to have proper HA.

Now, after all of that, what does this have to do with us? Well it has come to that point in the life of these sites what we need to move our services in house, our provider’s website does not provide the flexibility that we need to be able to bring everyone these great Q&A sites. Basically we have outstripped the capabilities of our provider’s web interface and the effort to code against their API is just not worth it compared to the added flexibility and benefit of running our own.

As part of bringing things in house I started looking for a good sizing tool, so that I could decide on the right hardware to support the amount of DNS traffic the Stack Exchange network receives. Unfortunately there is not a whole lot of information out there about sizing DNS servers. Luckily there are a lot of people that use our site who have a lot of varied experiences and knowledge – our very own Chris S provided me a rough formula to get a general Idea of the proper sizing for our DNS servers.

profile for Chris S. at Server Fault, Q&A for system administrators and IT professionals

His formula is fairly simple, and gives you a good idea of how much server you should buy (or if the equipment has already been purchased, how many requests it will be able to handle).

Chris’ formula:

req/s / 10 = CPU in MHz
Zone(s) file size * 2† = RAM (+OS)

†3 if dynamic updates are allowed

Over the next few weeks, we will be building out our own internal DNS system. We have the people and ability, and gosh darn it the desire to do DNS right. Pulling our DNS services in house will give use the ability to do things like DNSSEC, run Bind 10 when it stabilizes, and many other things to get us to our goal of being a great example to the world.


Kyle Brandt

It seems to me that every system administration community eventually de-evolves to a bunch of negative administrators that answers questions with things like:

> “RTFM.” > “Let me Google this for you ….” > “man ps, idiot.”

The next step is to end up like the typical IRC channel, “Happy to give you advice, as long as you don’t mind that I am going to insult you while I do it.” For those who haven’t spent time in many IRC channels it goes something like:

> “OMG You are SOOO STUPID. What is so god damn hard about calling a function by dereferencing a pointer to the function just because the pointer to the function is stored in a struct of array pointers that point to arrays of function pointers? , that was answered in comp.lang.c in `91 you moron. Didn’t you read the last 20 years of the usenet group, JESUS”

If it isn’t that bad, people just post a link without any explanation. The reason I love the Stack Overflow system is that is strongly discourages this. Well written answers will generally get upvoted above the bad ones. However there is a trait in our QA sites that not only does an upvote mean “You’re awesome, well written answer, and this is what I needed!” but it also can mean “I agree.” Herein lies the problem; if we all agree to de-evolve into a negative BOFH RTFM or the IRC attitude then the following happens:

Negativity = Frustration = “I am frustrated as well” = “I agree” = Upvote = Negativity gets upvotes

What we have here is a feedback loop or vicious cycle. Systems administration can be a frustrating job because often stuff simply doesn’t work. This is why so many system administrators end up frustrated and you get negative communities.

I am starting to sense that Server Fault might be inching in this direction. The reason is that short answers that are not that helpful are the slippery slope to RTFM. We can’t let this happen. Server Fault has an excellent system and is a great system administration community in a field that seems to me to be fragmented. The system is optimized to prevent sarcasm (for example, see Why are Stack Overflow People Nice?), but a system can only do so much against the will of a community.

A Case Study Today this came up in the Server Fault chat room when someone asked if the following was too sarcastic:

If you google “tail for windows” the first response is THIS, it’s called “Tail for Windows”.

Well it isn’t really sarcastic, but is isn’t the greatest answer either. Short answers that require someone to go hunting elsewhere is where the de-evolution in sysadmin communities starts. They may be helpful, but they really are not that helpful. If the link goes dead, then they are just no longer helpful at all. I have found the pattern on MSDN forums where all the posts are just a bunch of links to technet articles that you have to sift through which isn’t that useful.

Another thought was that “the author of the question hasn’t spent a single minute on google.” The most helpful answers on Server Fault are not about helping the person who asked, but rather they are about solving the question for everyone. I think the following is a better example of how to answer a question like that:

Have a look at Tail for Windows. It boasts the following features on their web page: A few features of Tail:
  • Watch multiple files in realtime
  • Detect keyword matches, and highlight occurrences
  • Send mail notifications on keyword matches by SMTP or MAPI
  • Plugin architecture allows you to write specialized handlers
  • Can process files of any size on all types of drive (local or networked)
  • Now this is by no means the best Server Fault has to offer, but that is not my point. The idea is that future visitors can see what features this program offers and decide if they want to try it. If there are other similar answers they have a comparison of programs that offer the functionality they need. One other thing to consider: If you are already taking the time to answer the question, making it more than just a link probably takes only about 10 seconds more of your time.

    What if a technet article has the answer the person needs? By all means link to it, but give a summary of what is in the article that you think answers their question. This saves time and makes sure the answer will always be useful. It also gives future visitors an idea if the link has the content they need. For example instead of just link:

    You can add service dependencies by adding the “DependOnService” value to the service in the registry using the `regedit` command, services can be found under `HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices`. The details can be found at MS KB article 193888, from which the following is an excerpt from: “To create a new dependency, select the subkey representing the service you want to delay, click Edit, and then click Add Value. Create a new value name “DependOnService” (without the quotation marks) with a data type of REG_MULTI_SZ, and then click OK. When the Data dialog box appears, type the name or names of the services that you prefer to start before this service with one entry for each line, and then click OK.”

    Here are couple things to keep in mind:

    1. Don’t think of answering the person who asked, think of answering the question for everyone. People will find it on Google in the future so really you answer the question for everyone.
    2. Any question you answer sets a tone for the site and our community. Good thorough answers encourage other people to do the same. Imagine a whole site of answers like the one you are writing — is it a good one?

    Keeping our answers excellent will create a tone for our community that attract other great administrators. That will lead to better answers for everyone and the bettering of our field.

    Intelligent Alerts

    Kyle Brandt

    The typical alert process for a typical system administrator goes like this:

    Ring Ring (Mr. Typical SA rolls out of his bed and looks at the phone.) (Stage Crew: Hold up 3×3 foot sign of alert: Critical: “Foo.baz.com is DOWN!”) Mr. Typical SA: “Balls.” (Mr. T SA walks over to his computer and starts to troubleshoot why his website is down) …

    The rest of the story you probably know. In summary, the SA diagnoses the problem and the site is back up in twenty minutes or so. He sends out an email explaining what happened and goes to bed feeling like a hero. If this sounds like you or your SA, congratulations, you have a competent system administrator who has just done is job right. No, really, I’m not being sarcastic, competent admins are hard to find. When I achieve competence I feel good about myself. Rightfully so, I think, everyone falls short from time to time and because this stuff is hard and doing your job well is a good thing.

    Beyond Competence Alerts and monitoring are a big part of system administration, and it is worth trying to explore how to go beyond being a competent administrator when designing a monitoring system. So looking at the above story, here is what happens:

    1. Alert: Tells the SA what has happened. Implemented by an automated monitoring system.
    2. SA diagnoses problem: This is the why, and if you are one step ahead of me, you may have noticed that in the above, this is in no way automated.
    3. SA fixes the problem.

    Good system administrators enjoy automating their tasks. Figuring out what the problem is, or at least starting to is a task the begs to be automated. So if an alert can not only tell what has happened but also why it happened then a system administrator is going beyond competence by designing an intelligent alert.

    Intelligent alerts that tell you “Why?” So what does it take for an alert that tells you why something has gone wrong? I think you need the following:

    • Monitoring all the layers that make your application work. This means things such as the network, sql, processes, cpu, memory, hits per second, etc. This way when you get what I call the front-facing alerts they are accompanied by back-end alerts. The front-end alerts is What and the back-end alerts are Why.
    • Log alerts are another key element. When something goes wrong admins generally check the logs as one of the first few steps. Log alerts can lead to skipping that step and since the alerts come at the same time and the automated temporal correlation can be very helpful.

    The tools to accomplish this are generally well known. For alerts there is Nagios and all those related tools and things like Splunk and its alternatives can be used for the log alerting. What this really comes down to a lot of work. It is a big time investment to get all these alerts going. My first post on this blog regarding fault tree analysis can help you brainstorm what you need to monitor.

    Predicting the Future The absolute best alerts will tell you about a problem before it actually happens. Probably the simplest example of this is to monitor disk space usage. When it has grown to certain percentage than you send alert. The admin gets the alerts before the server runs out of space.

    Where it can get interesting is when you don’t have something as simple as a fixed value. An example of an alert I set up in the past for this is the growth of mail queues in Exchange. What you generally care about here is not the size of the queue but if the queue has been growing. To do this I used samples from a rrd file from past times and compared it to current checks to calculate the percentage of growth. The rrd file was generated from Nagios performance data that was recorded for graphing purposes.

    Another example would be the amount of hits your web site is getting. What if you are getting a certain percentage of hits more for the time of the day, than you did the day before? Or what if you are getting a percentage of more hits for the time of day Monday that you did the last Monday? These sort of checks can predict a failure before it happens. Of course the big downside to predictive alerts is that is you don’t get credit for saving the day.

    For a sysadmin, there is nothing quite like the bliss of being able to get to your stuff no matter what. Now we can go to extremes with terminal emulators hooked up to serial concentrators, that are then hooked into each server as well as having a modem so we can dial in. Personally I think that is a bit over kill unless you are running systems that people’s lives depend on.

    Just like a good piece of cake, a proper Design-In-Depth remote access setup should be made of a few well selected complimentary system (or flavors depending on what you are currently working on!)

    There is Cake

    You should always be able to reach your servers somehow, even in the most extreme of circumstances. There are many technologies out there that allow you to manage servers remotely even when you can’t use the built in tools provided by the OS (ssh, RDP, etc).

    There are three pieces to building a data center that you can (almost) always get to.

    • Out Of Band Management (OOB)
    • IP KVM
    • Network Accessible PDU’s

    The most well known of these technologies right now is the OOB systems – most notable Dell’s DRAC and HP’s iLO system. These systems give you alot of control of the servers while they are not running an OS, but they fall short in a few areas, as well as can become unaccessable in certain situations.

    IP KVM’s are a complementary system to the OOB systems. They are appliances just like normal KVMs that sit in your rack and allow you to access multiple machines with one keyboard/monitor/mouse. The big difference is that they have a network port, as well as remote access clients so that you don’t need to be physically in front of the Keyboard/monitor/mouse that is hooked up to it to use it. IP KVMs fill in the gaps as well as give you a redundant path to your server console. As we all know redundancy is a Good Thing.

    I see you out there in the back of the auditorium … yes you waving you hand. You’re going to say

    >”But, they do the same thing! Why do I need to pay for both? I get OOB with my server and those IP KVMs are pretty expensive.”

    Well sir I have two answers for you. Firstly, this is part of design in depth, where you use complimentary technologies that run on different platforms to give you the most up time possible, as well as alternate paths to your equipment. Secondly, let me present a couple of scenarios to you.

    1. You have remote hands unpack and rack your brand new server. They hook everything up, but they cannot figure out your particular brand of OOB card. How do you go in and configure it yourself easily? Or how do you avoid spending 40 minutes on the phone walking them through setting up the card?
    2. It’s firmware update time of year! Yea! So you patch the OOB card’s firmware. when you reboot the server it doesn’t come back up and you can’t connect to the OOB card! How would you view the console? Oh and the server is 500 miles away in a Colo that only provides hands from 9-5 M-F and it’s now 3am on Saturday because you did this during a low traffic maintenance window. How would you get to the machine to see that the firmware update reset the IP on the OOB card and boot was stalled because the OOB card hung waiting for you to reset it?

    Your final line of defense is going to be network accessible PDU’s while they have many great qualities, the one that applies most to a remote access discussion is the ability to remotely power cycle the ports. If you every need to kick over a server that is half a continent away you will grow to appreciate these. Just make sure that you have your servers set to “start on power failure” in the BIOS.

    What is the moral of this story? Well it’s a concept I like to call design-in-depth basically the sysadmin equivalent to the security people’s concept of security-in-depth. You build a system that using different technologies that are complimentary to each other, and provide some overlap in functionality. This will allow you to someway, somehow fix what is broken right now. It also gives you options when you run into an issue with a particular technology. The more tools you put into your sysadmin toolbox, the better chance you have of achieving the goal of maximum up time with minimal user inconvenience.

    I am excited to announce that Stack Overflow is doubling the size the of the sysadmin team. We have picked up another member of the community and that member is George Beech. The regular users of Server Fault probably know him as Zypher:

    Server Fault profile for Zypher

    From looking at George’s answers on Server Fault and chatting with him in the Server Fault chat rooms I can tell you that George is a very knowledgeable sysadmin and also a cool guy that I am pumped to start working with. On Server Fault I have seen questions about server to admin ratios. I tend to agree that you can not prescribe a number. However, for a company with a popular web site I say at least two. I think all sysadmins should have a sysadmining buddy. Also between trying to exemplary sysadmins, writing blog posts, supporting employees, and dealing with the problems such as attacks on your website being a solo sysadmin is not ideal. Having two sysadmins is probably the most important place to have redundancy in your infrastructure. Lastly, a second person as great as George is going to provide all sorts of ideas and experience beyond what I could come up with.

    So congratulations George, I look forward to our adventure!

    When I started to build the the new data center I went with the Active Directory name stackoverflow.com. One advantage with this split DNS setup is that you can have internal IPs for certain entries. I knew I would have to duplicate domain entries but I did not mind that and I liked the aesthetic so I made that call.

    In most cases I don’t think it is really that bad but we happen to run a popular web site that uses that same name. If the developers have to make URL based calls to the site from the site than it would have to be worked around. Other downsides include that the duplicate entries could be wrong. The vote on Server Fault also seems to be against it. So on reflection, this was simply the wrong call.

    So yesterday I went about changing the AD domain name to a sub domain. I also was able to keep STACKOVERFLOW as the short name for the aesthetic. This has been a pain and I think there are still probably some things to fix but in the long run it will pay off.

    The moral of the story is that when you make a bad decision then suck it up and fix it. I think this is especially true at the core of a system. If something is wrong there than you have to work around that, and then you have to work around your workaround. What you end up with is:

    workaround = "workaround"
    while workaround:
        workaround += " the " + workaround

    So, if you can, just fix your bad decisions.

    I have spent the past couple of months getting our new data center in New York ready to start moving our production applications to. As I have built this out there have been two areas of administration which I have paid particular attention to:

    • Scalability of Management
    • Fault Tolerance

    Scalability of management means to me that I am ahead of the curve when it comes to my method of administration relative to the size of the infrastructure. So if there are 20 servers I want to administer them as if there were 100. The infrastructure right now is:

    • 10 Windows IIS Web Servers
    • 2 Windows MS SQL Database Servers
    • 2 Linux Routers
    • 2 Linux Load Balancers with HAProxy
    • 1 Linux Backup Server running Bacula
    • 1 Linux Management Server for Nagios and Logs
    • 2 VMWare ESXi Servers
    • 5 Dell Power Connect Switches

    It has been a lot of work getting all these operating systems installed, servers wired up (see the image below of one of two cabinets), and everything configured.

    However, I have made it into a lot more work that it might have been. Why? It wasn’t because I labeled each end of each cable as much as it is that I have aimed to build out everything in a way that the management of it will scale. More often than not it would have been faster to go and do everything manually on each server. However I have considered that bad for a few main reasons:

    If I gain the experience and knowledge of how to manage these servers as if there were at least 3 times as many I won’t have to deal with certain management issues when the pressure is on. I am sure that sort of growth will present all sort of other problems that I will be scrambling to solve. The analogy Jeff Atwood has used for the community problems we have faced as we have grown is big city vs. small city planning. I want to be as ready as possible to face the big city problems. I also believe in the long run this philosophy saves time. Lastly, manually administering each server as a single entity is just lame and pedestrian.

    Some of the tools I have used that I believe will aid in scaling my management are as follows:

    A Windows Deployment Server has allowed me to set up fully unattended installs of Windows for my servers. I was able to have 10 windows installs with all of them joined to the windows domain in just one morning. Of course it took me about a day to get the unattended configuration working after testing installs on VMs but in the long run I think it is worth it. Deploying servers as we grow will be that much easier.

    Group Policies for Windows has been essential to this aspect. These have allowed me to get much of the needed software installed either by using the msi installation feature or by setting up boot up scripts with them. This has included things such as enabling remote desktop, installing software packages such as NSClient++ and Mercurial, configuring firewall rules and deploying roles such as SNMP, IIS and .NET. The software installation aspect really can’t compare to something like apt for Debian but nonetheless it is essential to take advantage of what I can.

    Kludgy power shell scripts have been useful to for copying configuration files to all the servers and allowing me to restart the service. Although I have done a lot of bash shell scripting, power shell is new to me. However I was able to copy the NSClient ini file and restart the service using just a little snippet I picked up in not too long:

    $webservers = @("ny-web01", "ny-web02", "ny-web03", "ny-web04", "ny-web05", "ny-web06", "ny-web07", "ny-web08", "ny-web09")

    foreach ($server in $webservers) { Write-Host $server cp 'C:Program FilesNSClient++NSC.ini' "\$serverc$Program FilesNSCLient++NSC.ini" $svc = Get-WmiObject Win32_Service -ComputerName $server -Filter "name='nsclientpp'" if ($svc.started -eq $true) { $svc.StopService() } $svc.StartService() }

    This as well had a higher up front time cost than just manually copying the file, but in the long run it is better administration, it scales, and saves time. I have already had to make some changes to that configuration and pushing the changes now is fast and simple.

    I have deployed a Windows Update Server (WSUS) to allow me to control the updates that go to certain servers from a centralized location. I opted to have the updates are scheduled via a group policy and controlled with WSUS by using client-side targeting.

    Nagios has also been essential in this philosophy in that it uses an inheritance and template system. Because of this once all the services and groups are configured all I have to do is add the host the group to getting everything monitored. To further extended this I have used n2rrd for graphing which plugs directly into Nagios. This way adding checks and graphing to a new server deployment is all done in a single step. Also if I want to add a new check to all the servers once I get it setup I just have it run for the entire group which is as simple as adding it the host group. Gathering the data in the graph means storing the results over time which is essential for trending so that new hardware can be purchased ahead of time.

    All these tools are the tools I think that are needed to build a city. Some tools are good for a server or two but they just don’t scale.

    There is still more to do. For the Linux boxes I have been using ssh and loop scripts similar to the power shell script above and really something like Puppet is more appropriate. Also not every piece of software had a simple way to deploy via a group policy. Sadly enough, .NET 4 doesn’t come as an MSI and isn’t available in the update services yet (but is coming “soon”, as in 2-12 months). Also a centralized logging solution is on my to do list as checking the logs on each server is quite tedious.

    Fault Tolerance: Fault tolerance is basically the idea that if one component fails it doesn’t bring everything down. I am of the belief the network is a good area to start with fault tolerance and this is where I have put much of my efforts.

    Network Fault Tolerance: In a previous Server Fault Blog post entitled “HSRP is not for WANs” I explained why I didn’t think HSRP was valid for the WAN and that a routing protocol should be used. I managed to convince our collocation provider Peer 1 of this and we now have a private BGP peering with them. On our 2 Linux routers I use the Quagga routing suite for BGP. ucarp is used to provide a redundant gateway for the LAN. For stateful firewall rules in IPTables I use conntrackd to make sure that any asymmetric routing patterns don’t break because of mismatched state tables.

    Each server is connected to two different switches and has bonding in an active fail over mode.

    With all of this going on the key has been to test the fail over by bring down the routers and interfaces on the routers. Without the tests you are only hoping you have fault tolerance.

    Power Fault Tolerance in Thirds: Power can be tricky for fail over. Many people think you just put two power supplies in each server connected to independent feeds and you are good to go. However, I had learned from others experiences that at least with the older Dell PowerEdge servers you didn’t know which power supply would be active. If you don’t properly watch your power when one of the feeds fails you can overload the other and have no power. So the alternative I have taken is to divide the servers into groups of 3 (when there are more than two). One server will have dual power supplies, the second will be connected to the A feed and the third server to the B feed. This way if power fails we will be operating at 66% instead of 50% capacity. In general when there are only 2 servers the second is a warm backup so I just make sure they are on different feeds. If there is only one server that I have dual power supplies in it).

    Fault Prevention: I have also made sure to set up a staging server in the production environment. The best way to handle faults is to stop them from happening and I think having a test server should cut down on them quite a bit.

    I think scalability of management and fault tolerance are just two aspects that make the difference between administering servers and being a professional system administrator. These are the sort of topics that make system administration a profession and it is one reason why Server Fault can be such a great resource for system administrators.

    Server Fault now has a place for real time chat over at chat.serverfault.com. We have had a room going for a few weeks now over at meta.chat.stackoverflow.com but it is nice for us to have our own place now where we can create specific chat rooms now.

    A place for live chat is a great thing to have for system administrators. A lot of system administrators seem to be out working on their own so this is a nice place to socialize. It also can be a place to talk about those questions that don’t really have straightforward answers. For example:

    If you haven’t seen the chat system that Stack Overflow’s employees Benjamin Dumke and Marc Gravell have developed you are missing out. I bet a lot of system administrators, especially grumpy ones with a *nix slant like me, might have a first thought similar to what I had:

    Why do we need some Web 2.0 garbage to replace our wonderful text based IRC?

    It turns out there a lot of reasons besides all the damn netsplits. This chat retains all the history so when you join a room you can see what was being talked about, you can also search the transcript. There is a lot more as well — I recommend you go check it out.