We have been live at the New York data center with stackoverflow.com, superuser.com, serverfault.com and the rest of the Stack Exchange network for a work week now and the SysAdmin team is pleased to say that it has gone well. It has been smooth since launch with the only downtime being for some quick tweaking we wanted to try.

So how does the performance look when it comes to our improved latency promise?

We use Pingdom to monitor our servers from outside our network. This data is the average time to load the Stackoverflow.com homepage from all the Pingdom locations which includes locations throughout Europe, The United States, and Canada to load the Stackoverflow.com homepage. Given, this is not exactly scientific, but the big drop in starting the date of the move (October 24th) is a big win in my book. When I, well, the Pingdom web interface, ran these numbers for the Texas and Chicago I also saw an improvement in the latency.

Diagrams Please
When we got an email of someone offering to do a network diagram for us I knew it was time to get something up:

This is just an overview of the network — on the left is our public facing network and the right side is our private network. There might be some reworking of the network layer down the road (more on that later).

The Web Servers:
We now have 4 more web servers which are also more powerful. These are all Windows Server 2008 R2.

Old Servers:

• Lenovo ThinkServer RS110
• 1x Intel Xeon X3360 2.83 GHz quad-core CPU
• 8 GB of RAM

New Servers:

• Dell R610
• 1x Intel Xeon Processor E5640 2.66 GHz Quad Core with 8 threads.
• 16 GB Ram

We now keep all the web server configs the same so adding another server for a site is just a load balancer change. So currently stackoverflow.com runs on 6 servers (3 servers dedicated for stackoverflow only).

The CPU graphs on our web servers show just how much of better performers they are for us. The following is a comparison of dedicated stackoverflow.com web servers (Keep in mind that these are on 6 servers instead of 3 now).

Old:

New: In general the CPU load on our web servers has been balanced and it still is. In short, we have plenty of room for growth here.

The Database Servers:
Our database servers are also more powerful. Old Database Servers:

• Lenovo ThinkServer RD120 barebones
• 2 Intel Xeon E5420 2.5 GHz quad-core CPU
• 48 GB of RAM
• 6 SPindles

New Database Servers:

• Dell R710s
• 2x Intel Xeon Processor X5680 3.33 GHz
• 64 GB of RAM
• 8 Spindles

We are currently working through a full text performance regression (Possibly going from SQL 2008 to R2) and will upload graphs when we have a more meaningful comparison.

The Network Layer:
Our router/firewall are both Linux machines with the same basic specs as the web Tier but with less memory. My goal has been perfect redundancy at the network layer and to eliminate all SPOFs. To this end we have two switches, CARP on our LAN and a private BGP peering with our provider, redundant switches and bonded nics. I got close but tests show that I have failed. Right now at the heart of this problem there are two things:

1. Quagga just doesn’t work the way I would expect dynamic routing protocols to work. If a route disappears it keeps announcing it unless I redistribute static routes. Also, if a connected route goes down and it also knows about the route through iBGP it doesn’t insert the iBGP route into the table but does remove the static route.
2. Stateful Firewalls and asymmetric routes don’t like each other. The fundamental problem is that they would need to sync state faster than SYN and then the SYN ACK or suffer performance problems. There is an interesting whitepaper on this topic: “Demystifying cluster-based fault-tolerant Firewalls.”

George and I were left with two options, try to force symmetry in the entire network or eliminate statefulness. After a day of back and forth we have decided that Linux/Quagga isn’t working for us and that we also need to separate out our router and firewall levels. Our plan is currently to move to Cisco because we are familiar with them and the time cost to explore Juniper or BSD is not appealing to us at the moment. Currently however we can lose either the router our either WAN link and still automatically come back up. So do have redundancy, it just isn’t perfect yet.

Redis and our Load Balancers:
These are also currently spec’ed out the same as our web tier machines except that the load balancers have 4 GB or RAM. They run Linux (Currently a mix of Ubuntu Server and CentOS). If you have been following this blog you know we use HAProxy and love it. We also use Redis as our distributed caching layer.

Internal Network:
Besides the database our Internal network has the following:

• A Linux backup Server running Bacula which also handles our web logs and web log analysis (R610, 32 GB Ram, Xeon E5640)
• A Linux monitoring Server with Nagios and Splunk and n2rrd for Graphs. (R610, 32 GB Ram, Xeon 5640)
• 2 ESX Servers for our Domain Controllers (R610, 16 GB Ram, Xeon 5640)

From a sysadmin perspective, all that ram gives us the ultimate log analysis playground real estate.

What is next?
We still have to move our sstatic.com domain. George is currently building out our in house external facing DNS servers. We also still have our Oregon data center so some geographic load balancing of the static content is on our wishlist. Also, we are trying to gather all the stats we can into Nagios so we can measure the impact of our tweaks, trend, solve problems quickly, and enjoy the info porn.

We scale in three directions up, out, and awesome people. We don’t have lots of small weak servers (out) nor do we have a couple of monsters (up) — rather we strike a balance. Our third, and most import scaling direction is getting awesome people. By that I mean we have great programmers who happen to be performance junkies. They literally tune the code 24/7 and also know a lot about SQL and system tuning as well. The sysadmin team will be working with them to get the best damn performance we can from our new machines.

We are starting the move of our services from our Oregon Data center to Our Brand-Spanking-Shiny-New NYC Data Center.

We’ll do our best to keep things up and running while we move but there may be some service interruption and oddities as we get everything over and running.

We should have all the migrations done sometime today (but sooner than 6-8 weeks)

Check back here for updates on the move.

DNS is one of those systems that is one of the easiest to setup, yet it is one of the hardest to setup right. Most of the time just works is good enough for probably 90% of the installations out there. But as system administrators, just works should never be what we strive for. We should always go for the best implementation that we possibly can achieve.

There are basically two choices you need to make when you want to setup your DNS.

• Do I want to host DNS internally
• Do I want to host DNS externally

Both of these options have their advantages and drawbacks. In broad strokes they can be listed as:

Advantages

Hosted Solution’s

• Low management overhead – you don’t need to worry about the hardware, high availability and distribution of server
• Low learning curve – You are generally given a web interface that makes it very simple to add records, with explanations for what each of them do to help you along
• Low cost to get in, on the low end of traffic levels very affordable.

In House

• Highly customizable
• You aren’t reliant on another entity for your DNS
• When your traffic starts to climb, the cost/benefit ration is squarely in the favor of in house hosting

Disadvantages

Hosted Solutions

• At a certain volume of traffic they can become very expensive
• Making many changes can be painful without writing code against their API, if they have one
• You are reliant on another entity to keep a core service running

In House

• You have to put out the hardware cost up front
• You need to have people with the proper skills to manage the servers properly
• You need to have the a large enough infrastructure to have proper HA.

Now, after all of that, what does this have to do with us? Well it has come to that point in the life of these sites what we need to move our services in house, our provider’s website does not provide the flexibility that we need to be able to bring everyone these great Q&A sites. Basically we have outstripped the capabilities of our provider’s web interface and the effort to code against their API is just not worth it compared to the added flexibility and benefit of running our own.

As part of bringing things in house I started looking for a good sizing tool, so that I could decide on the right hardware to support the amount of DNS traffic the Stack Exchange network receives. Unfortunately there is not a whole lot of information out there about sizing DNS servers. Luckily there are a lot of people that use our site who have a lot of varied experiences and knowledge – our very own Chris S provided me a rough formula to get a general Idea of the proper sizing for our DNS servers.

His formula is fairly simple, and gives you a good idea of how much server you should buy (or if the equipment has already been purchased, how many requests it will be able to handle).

Chris’ formula:

req/s / 10 = CPU in MHz
Zone(s) file size * 2† = RAM (+OS)

†3 if dynamic updates are allowed

Over the next few weeks, we will be building out our own internal DNS system. We have the people and ability, and gosh darn it the desire to do DNS right. Pulling our DNS services in house will give use the ability to do things like DNSSEC, run Bind 10 when it stabilizes, and many other things to get us to our goal of being a great example to the world.

It seems to me that every system administration community eventually de-evolves to a bunch of negative administrators that answers questions with things like:

> “RTFM.” > “Let me Google this for you ….” > “man ps, idiot.”

The next step is to end up like the typical IRC channel, “Happy to give you advice, as long as you don’t mind that I am going to insult you while I do it.” For those who haven’t spent time in many IRC channels it goes something like:

> “OMG You are SOOO STUPID. What is so god damn hard about calling a function by dereferencing a pointer to the function just because the pointer to the function is stored in a struct of array pointers that point to arrays of function pointers? , that was answered in comp.lang.c in `91 you moron. Didn’t you read the last 20 years of the usenet group, JESUS”

If it isn’t that bad, people just post a link without any explanation. The reason I love the Stack Overflow system is that is strongly discourages this. Well written answers will generally get upvoted above the bad ones. However there is a trait in our QA sites that not only does an upvote mean “You’re awesome, well written answer, and this is what I needed!” but it also can mean “I agree.” Herein lies the problem; if we all agree to de-evolve into a negative BOFH RTFM or the IRC attitude then the following happens:

Negativity = Frustration = “I am frustrated as well” = “I agree” = Upvote = Negativity gets upvotes

What we have here is a feedback loop or vicious cycle. Systems administration can be a frustrating job because often stuff simply doesn’t work. This is why so many system administrators end up frustrated and you get negative communities.

I am starting to sense that Server Fault might be inching in this direction. The reason is that short answers that are not that helpful are the slippery slope to RTFM. We can’t let this happen. Server Fault has an excellent system and is a great system administration community in a field that seems to me to be fragmented. The system is optimized to prevent sarcasm (for example, see Why are Stack Overflow People Nice?), but a system can only do so much against the will of a community.

A Case Study Today this came up in the Server Fault chat room when someone asked if the following was too sarcastic:

If you google “tail for windows” the first response is THIS, it’s called “Tail for Windows”.

Well it isn’t really sarcastic, but is isn’t the greatest answer either. Short answers that require someone to go hunting elsewhere is where the de-evolution in sysadmin communities starts. They may be helpful, but they really are not that helpful. If the link goes dead, then they are just no longer helpful at all. I have found the pattern on MSDN forums where all the posts are just a bunch of links to technet articles that you have to sift through which isn’t that useful.

Another thought was that “the author of the question hasn’t spent a single minute on google.” The most helpful answers on Server Fault are not about helping the person who asked, but rather they are about solving the question for everyone. I think the following is a better example of how to answer a question like that:

Have a look at Tail for Windows. It boasts the following features on their web page: A few features of Tail:

Watch multiple files in realtime

Detect keyword matches, and highlight occurrences

Send mail notifications on keyword matches by SMTP or MAPI

Plugin architecture allows you to write specialized handlers

Can process files of any size on all types of drive (local or networked)

Now this is by no means the best Server Fault has to offer, but that is not my point. The idea is that future visitors can see what features this program offers and decide if they want to try it. If there are other similar answers they have a comparison of programs that offer the functionality they need. One other thing to consider: If you are already taking the time to answer the question, making it more than just a link probably takes only about 10 seconds more of your time.

What if a technet article has the answer the person needs? By all means link to it, but give a summary of what is in the article that you think answers their question. This saves time and makes sure the answer will always be useful. It also gives future visitors an idea if the link has the content they need. For example instead of just link:

You can add service dependencies by adding the “DependOnService” value to the service in the registry using the `regedit` command, services can be found under `HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices`. The details can be found at MS KB article 193888, from which the following is an excerpt from: “To create a new dependency, select the subkey representing the service you want to delay, click Edit, and then click Add Value. Create a new value name “DependOnService” (without the quotation marks) with a data type of REG_MULTI_SZ, and then click OK. When the Data dialog box appears, type the name or names of the services that you prefer to start before this service with one entry for each line, and then click OK.”

Here are couple things to keep in mind:

1. Don’t think of answering the person who asked, think of answering the question for everyone. People will find it on Google in the future so really you answer the question for everyone.
2. Any question you answer sets a tone for the site and our community. Good thorough answers encourage other people to do the same. Imagine a whole site of answers like the one you are writing — is it a good one?

Keeping our answers excellent will create a tone for our community that attract other great administrators. That will lead to better answers for everyone and the bettering of our field.

The typical alert process for a typical system administrator goes like this:

Ring Ring (Mr. Typical SA rolls out of his bed and looks at the phone.) (Stage Crew: Hold up 3×3 foot sign of alert: Critical: “Foo.baz.com is DOWN!”) Mr. Typical SA: “Balls.” (Mr. T SA walks over to his computer and starts to troubleshoot why his website is down) …

The rest of the story you probably know. In summary, the SA diagnoses the problem and the site is back up in twenty minutes or so. He sends out an email explaining what happened and goes to bed feeling like a hero. If this sounds like you or your SA, congratulations, you have a competent system administrator who has just done is job right. No, really, I’m not being sarcastic, competent admins are hard to find. When I achieve competence I feel good about myself. Rightfully so, I think, everyone falls short from time to time and because this stuff is hard and doing your job well is a good thing.

Beyond Competence Alerts and monitoring are a big part of system administration, and it is worth trying to explore how to go beyond being a competent administrator when designing a monitoring system. So looking at the above story, here is what happens:

1. Alert: Tells the SA what has happened. Implemented by an automated monitoring system.
2. SA diagnoses problem: This is the why, and if you are one step ahead of me, you may have noticed that in the above, this is in no way automated.
3. SA fixes the problem.

Good system administrators enjoy automating their tasks. Figuring out what the problem is, or at least starting to is a task the begs to be automated. So if an alert can not only tell what has happened but also why it happened then a system administrator is going beyond competence by designing an intelligent alert.

Intelligent alerts that tell you “Why?” So what does it take for an alert that tells you why something has gone wrong? I think you need the following:

• Monitoring all the layers that make your application work. This means things such as the network, sql, processes, cpu, memory, hits per second, etc. This way when you get what I call the front-facing alerts they are accompanied by back-end alerts. The front-end alerts is What and the back-end alerts are Why.
• Log alerts are another key element. When something goes wrong admins generally check the logs as one of the first few steps. Log alerts can lead to skipping that step and since the alerts come at the same time and the automated temporal correlation can be very helpful.

The tools to accomplish this are generally well known. For alerts there is Nagios and all those related tools and things like Splunk and its alternatives can be used for the log alerting. What this really comes down to a lot of work. It is a big time investment to get all these alerts going. My first post on this blog regarding fault tree analysis can help you brainstorm what you need to monitor.

Predicting the Future The absolute best alerts will tell you about a problem before it actually happens. Probably the simplest example of this is to monitor disk space usage. When it has grown to certain percentage than you send alert. The admin gets the alerts before the server runs out of space.

Where it can get interesting is when you don’t have something as simple as a fixed value. An example of an alert I set up in the past for this is the growth of mail queues in Exchange. What you generally care about here is not the size of the queue but if the queue has been growing. To do this I used samples from a rrd file from past times and compared it to current checks to calculate the percentage of growth. The rrd file was generated from Nagios performance data that was recorded for graphing purposes.

Another example would be the amount of hits your web site is getting. What if you are getting a certain percentage of hits more for the time of the day, than you did the day before? Or what if you are getting a percentage of more hits for the time of day Monday that you did the last Monday? These sort of checks can predict a failure before it happens. Of course the big downside to predictive alerts is that is you don’t get credit for saving the day.