This topic was suggested by one of our users, Bart Silverstrim. He was curious about how companies could maintain an environment where maintenance downtimes would not actually affect customer experience. Think of large sites like Google, Facebook, CNN, pretty much any site in the quantcast or alexa top 100. These sites maintain extremely good uptime numbers, but how do they do it?

It’s no secret that sites like this employ a lot of servers to handle content delivery. There are not only load balanced server clusters but also CDNs and caching proxies that help mitigate some of the load on the environment. Eventually, though, all of these machines are going to need maintenance at some point. How does one do this and not affect perceived uptime from the users? This question is actually rather complex because it depends on multiple ecosystems to properly execute.

Monolithic Deployment vs. Continuous Deployment

For many years, the method of software development followed a pretty static and time consuming process:

1. Define scope for this release
2. Program features for the release
3. Debug, if errors, back to #2
4. QA (generally someone testing who ISNT the developer), if errors, back to #2
5. Acceptance/Staging
6. Deployment

Now, I know a lot of shops that would laugh at this list and claim that they do only #2 and #3, then shove it up to the servers. This can very well be the case, but I’d hazard a guess that those companies have never gotten close to scraping the alexa top 100, nor would I believe with a straight face if they told me they had uptimes in excess of 99%. There is another way, though: Continuous Deployment.

Simply put, Continuous Deployment is when instead of having large, multi-bug/multi-feature releases, you instead implement many, many small changes continuously throughout the product lifetime. At first glance this method might sound dangerous to many sysadmins who like to plan a task to an atomic point algorithmically, but it’s actually extremely safe if you follow some standard methodologies, namely having an environment where you can test changes (for Stack Exchange, this is meta.stackoverflow.com) and use a build proctor like CruiseControl or TeamCity to facilitate the build process.

One of the main benefits of Continous Deployment is the fact that small changes are easy to deploy and usually just as easy to revert should problems arise. It becomes significantly harder to revert a deployment if there is a large corpus of changes contained in the update.

Code it like you’ll need to change it

One thing that new programmers often fall prey to is the laziness of not properly architecting their application. I am sure that every programmer remembers their earlier code shenanigans when they’ve needed to update something they wrote earlier in their career only to find that it was fraught with all sorts of freshman mistakes. To put it simply: you need to write your code to be portable, easily understood, and utilizing the best practices available for that language. If the language is OO, this means obeying standards like MVC, creating class interfaces and making the app as dynamic as possible. The reason should be clear: code that’s easier to interface with is easier to change and update, and it generally means that bugs introduced in most modules should not have global impact to the site.

Load Balancing is Good

Load Balancing is paramount to a seamless customer experience. It allows you to do some pretty cool tricks, but only so long as your web application is coded to properly handle a load balanced environment. The main thing one needs to think about when coding an app in a load balanced environment is that there’s no guarantee that request ‘n+1’ is going to land on the same server as request ‘n’, so you need to handle sessions in a centralized/db manner so that the cookies in the browser link you up to the proper session regardless of what server you hit. This does NOT mean, however, that you should disable persistance or affinity in your load balancer! There are benefits to keeping a session on the same server with regards to file caching and the like; we just want to make sure the app is ready should you want to take one of the servers down for maintenance.

Once properly load balanced, you gain several levels of win. First off, your app will be much more performant if there are more servers available to handle the load. Secondly, if you have multiple servers available, you can bring one of them offline and your users should never know the difference. A side benefit of Load Balancing is the peace of mind you’ll get knowing that if a box eats itself alive at 3am, your site will survive without you needing to fire up the laptop or, heaven forbid, head into the office at an extremely early hour.

How Stack Exchange does it

It may become clear after reading this that what I’m talking about isn’t necessarily how Stack Exchange works. People who visit us often know that we do have site downtimes for various reasons. For those curious, the below section is how we handle our deployment and development process.

There is one big place where the Stack Exchange Core Q&A Services are vulnerable, and this is at the database. Currently we employ SQL 2008 R2 database servers (currently SP1 as of this writing), with the primary server constantly replicating to the backup server via transaction log shipping. Those familiar with transaction log shipping will know that employing this method is really only usable in an active/”hot passive” mode. One of the downsides of using SQL Server is the lack of solid high availability given our transaction load. Simply put, both the asynchronous and synchronous methods of active/active can’t keep up with the sheer volume of transactions we throw at it. We’re hoping that when Denali comes out this spring, the HA features will improve to the point where we can be fully active/active.

The reason we incur downtimes for upgrades these days has to do partially with ease of execution and cleanliness; because we use transaction log shipping, if we wanted to go active on the backup node we’d have to break the replication and convert the backup server to the master server, then re-setup replication. Would we do this for a couple windows patches? Absolutely not. It’s a nontrivial amount of work that can incur human error and is unnecessary. We reserve this process for when we do have a major database emergency.  We’re hoping that SQL 2012’s enhanced failover capabilities will permit us to enter read-only mode for brief maintenance windows, but this will need serious testing first, once Denali (2012) is released.

Development wise, this is the continuous deployment procedure that Stack Exchange uses:

1. A developer will get the latest HEAD from the mercurial repository.
2. That developer makes a change, then commits that change back to the Mercurial repository.
3. TeamCity queries Mercurial (every 60 seconds) to check for new changes. If a change is found, TeamCity builds it immediately in the development testing environment.
4. Once the developers have given the change a test in development, the developer then deploys the change to meta.stackoverflow.com.
5. If the dev is having a good day, meta.so’s users won’t report any bugs and after a period of time the dev will push the code to everything except stackoverflow.
6. If the change stands up to rigorous testing on the other approximately 80 sites in our network, the change will be pushed to stackoverflow, aka “the fire hose.”

One might be curious as to why we wait till the last minute to push code changes to stackoverflow. The reason behind this has to do with the fact that stackoverflow gets several orders of magnitude more traffic than the rest of Stack Exchange combined. A case study in this procedure: when we deployed ProtoBuf v2, the change worked great everywhere else, but as soon as the change was applied to stackoverflow.com, a “cold start” bug seriously degraded users’ experience.

An important thing to note as well is that a great deal of the sites’ code changes are toggleable by configuration, so if a problem is found it can be reversed and mitigated much, much faster than it would take a developer to debug and fix the problem.  Employing this method where possible in your applications will be helpful for many reasons.

Uptime! Fsck Yeah!

For most of our readership, the problems described above might not apply to you. You may not have quite as many hits as Stack Exchange, or you might have less need for high availability and uptime. This isn’t to say that the above doesn’t apply; architecting for uptime should be in every developer’s best interests and best practices.

As always, your questions and comments are most welcome, feel free to post below.

If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

• Placing a value on performance: “Performance is a feature”
• Well integrated development and operations
• A sense of craftsmanship when it comes to performance

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent.  This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”

Monitoring is an extremely important tool in your arsenal of preventative measures.  Monitoring is important for a great deal of different reasons:

• Monitoring allows you to send alerts if certain conditions are met,
• It allows you to visualize trends in data,
• Provides a method of assurance to the customer that their consumed services are guarded,
• Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.

Your monitoring environment need not be this complex to be useful.

There are many different products in the monitoring sphere.  Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use.  My personal favorite is Nagios, though it does have some shortcomings that I will touch on later.  Most monitoring systems follow the same basic recipe:  You configure hosts, which in turn have services or metrics you want to monitor.  The monitoring system will “optionally” alert you if you configure it to do so.  Most monitoring systems have a method of keeping historical data and graphing it.  This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.

There are several methods of monitoring.  The most basic and least useful method alone is a simple ping test.  Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert.  Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics.  This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see.  SNMP does have its benefits, though:  since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box.  It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.

Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses. For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe monitoring to advanced Microsoft SQL statistics checking.  In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI.  Being able to do this vastly improves the metrics you can extract from windows servers.  At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell.  This might have only been related to the Exchange family of products, though, so don’t take it as gospel.  Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things.  WMI is an instrumentation interface, whereas PowerShell is a scripting language.  It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)

A glimpse behind the curtain: this is one of the primary StackExchange monitoring pages.

One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem.  Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems.  Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware.  I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party.  Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring.  Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.

A helpful part of many monitoring systems is being able to group hosts and services into logical containers.  This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group.  If your monitoring system supports grouping and you are not using it “you are doing it wrong.”

Do you need a monitoring environment?  Yes.  There is no other answer to this question.  If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system.  The only question is, how much do I have to monitor?  This depends a lot on your customer SLAs and the expectactions of uptime.  As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”

Having a lot of stuff monitored also helps with correlation and causation.  For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error.  Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly.  If you were just monitoring the web server, you just lost thirty minutes.  If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server.  All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.

Even Xzibit agrees, more monitoring is better monitoring.

One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox.  Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active.  There are a few ways to solve this.  First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems.  If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system.  You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time.  Regularly auditing the monitoring environment is the best way to keep your faith in the system.

In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one.  Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers.  Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.

Welcome back to my series on WiFi. In Part 1 of the series, I began with some basics of RF and explained some differences about antennas. It should be apparent at this point that there is a science behind this activity, and I’ll take this moment to warn you thoroughly before we move on: These posts are a good way for you to become familiar with WiFi and should provide you with some solid knowledge to help improve your WiFi coverage. However, this brief education is not a replacement for having an actual RF engineer do a site survey of your environment! If you have a “must work right the first time” environment, and you’re reading this because you’re the decision-maker and don’t have the slightest hint about what all this is about, Get An Expert. They do this all day long. It’s money well spent.

If you do use these techniques below, Your Mileage May Vary. It’s also important to note that if you go to all this work, setup your access points, then your neighbor goes and installs his AP right next to yours on the same channel, then you’re going to be stuck re-doing these activities all over again. WiFi isn’t a static situation; as people get more WiFi-connected devices, the playing field changes, and it will change on you, I guarantee it.

Understanding RF Interference and What It Does to WiFi

You hear people joke about microwave ovens interfering with WiFi equipment pretty often. Most people laugh it off as an urban legend. It’s not. Below, I have included some RF spectrograms for your entertainment. If you haven’t seen images like this before, they are a visualization of signal frequency and intensity over time. Past-to-present is a top-to-bottom relationship, and the colors are a heatmap (with red being a strong signal.) As you look at both types of graphs, the channels start from 1 at the left hand side of the graph, and go up through 12 in the right hand side. NOTE: Quick Shout-Out to the guys at metageek.net for creating the awesome Wi-Spy and accompanying Chanalyzer Pro software. We paid full price for the DBx bundle (Comes with the Wi-Spy DBx and Chanalyzer Pro) and I definitely feel like it was money well spent. Check them out if you want to do these types of visualizations yourself.

These images show what the wireless signal looks like in my suburban home. Not a lot of interference in this visualization, you can see my home Cisco Aironet 1240 AP humming along happily as visualized by the wavy lines in the waterfall spectrogram, above. In the lower graph, we see signal strength (Amplitude) measured by frequency.

Let’s shake things up, and show what happens when you fire up a microwave oven:

Look out, here comes that microwave burrito exploding all over your RF Spectrum! For about 30 seconds, I nuked a mug of water and this was the result. You can see through the swamp of RF that the access point does its best to compensate for the signal interference, but that’s a pretty strong blast of RFI.

Do you have a baby monitor at home? Is it on 2.4ghz? Ready to see what it’s doing to your wireless signal?

These two charts were captures I took from my friend’s house (incidentally, the gentleman who I mentioned in the previous post — he has a penchant for wifi problems.) I was over his house and ran some traces to get a visualization of his wireless conditions in preparation for installing a new wireless router. I asked him if he noticed the WiFi being slower at night and he’d mentioned that it did indeed seem to be more problematic at night. Welcome to the wonderful world of random equipment in your home causing issues with your wifi. In the above trace, you can see the telltale wavy lines of the access point, trying to power its way through the interference. The graph below has just the slighest hint of bell curve, which is where his AP was situated in the RF Spectrum. I believe in this case his AP was on channel 3. Needless to say, we popped his new wifi router on channel 11, which is quiet in these graphs.

One final graph to show. If you scroll back up to the initial image I showed of my suburban home, this will give you an idea of what your general household’s 2.4gHz spectrum might look like. Now, compare that image to this:

This, my friends, is what the 2.4gHz spectrum looks from the Stack Exchange offices. We’re located down by Wall Street, on the 26th Floor of One Exchange Plaza. Our scenic vantage point does come with a cost! These spectrograms show just how much RF interference we are subject to at this location. Astute readers may notice the timescale difference on the graphic, but I assure you that the 30 second view is just as nasty.

What can we take away from these charts? One could safely summarize this entire section as “Location and the gadgets in your home both play a significant role in how your WiFi might perform.”

Mapping Your Wireless Landscape

It’s worth noting that even though the above charts were taken using a very expensive measuring tool, your laptop’s WiFi card is a potent ally in your quest to improve your coverage. For the next section, I am assuming that you firstly are running Windows and have downloaded and installed both Vistumbler as well as Microsoft SQL express. I am aware that a strong number of our readership are Linux based, and there is also a big Mac contingent. I’ll unabashedly say that the steps I’m following and the software choices I made were purely for my convenience, but I hope that I explain the process in easy enough terms so that the tinkerers out there can take the wheat from the chaff, so to speak.

Step 1 – Take some measurements!

Vistumbler is a wardriving utility that, when attached to a gps, can help you map where there are wireless access points in range of your device. We’re going to borrow it for a more sedentary purpose. Fire up Vistumbler, set your laptop in the areas where you want to consume your WiFi signal, and then start the scan/capture process. Leave the laptop there for at least 30-60 minutes, as we want a whole lot of datapoints to work with. It will keep track of every time it hears of an access point and record the relative strength of the signal. Once the time is up, you can either run the detailed export to CSV now, or “Exit (Save DB)” and come back to export the file later. NOTE: If you’re in a tight urban region like we are at Stack Exchange, leaving Vistumbler up for 30+ minutes will result in a tremendous amount of data! It’s wise to have a very powerful PC to handle the vistumbler export process, or do seperate scans and aggregate the data together in a later step.

Step 2 – Massage the data!

Sadly, Vistumbler’s export to CSV does leave a bit to be desired with its field quoting. We’re going to open the csv in Excel, since it seems to be especially forgiving. Once we’ve opened it in Excel, we’ll do the following steps:

• Make a new column for Location. Populate the column with a location name. You’ll want this when you’re querying SQL later on.
• Save the file as an Excel spreadsheet.
• Fire up SQL Server Management Studio.
• Create a new database if you don’t already have a good scratch database, then right-click the db and select tasks->import data.
• Using “Microsoft Excel” as a datasource, submit your new excel file as the source data. HINT: If you’re getting an error about unable to find a particular OLE provider, and you’re on 2010 like me, with 64bit windows, you will likely need this link to load said provider.
• Select the destination database; elementary stuff here. From this point on, “Just Keep Hitting Next,” except for the prompt where you specify the destination table name. I strongly advise you to change that name to something easier to type rather than the default date/time string. Finally, Finish to start the import job. This may take a while, so don’t be afraid if it seems like it’s taking too long.
• Repeat these steps for each file of data. Be sure to specify the same table name for each import, it will append to the database.

Step 3 – Analyze!

With this complete, congratulations! You now have data in a sql server that you can use to leverage the power of SQL to get some statistics from. I’ll admit I’m a SQL neophyte — I can do some joins and “GROUP BY”s but I’m sure others could tease a lot more information out of it than I have. Here are some basic queries for your dataporn pleasure:

Get a sorted list of strongest access points across all locations:

select LOCATION, SSID, AVG(SIGNAL) AS AVGSIGNAL
FROM [dbo].[wifilog]
GROUP BY LOCATION,SSID
ORDER BY AVGSIGNAL DESC

In the above query, we see that in our sysadmin office, as well as somewhere in our office (for shame, I forget where I took the trace!) the strongest signals are SO-GUEST (our current guest wireless AP) and ROVIO (for our cute little mobile webcam.) We’ve also got a couple shadow AP’s (as specified by NULL) followed up by some other AP’s that I’m not sure who or where they are. Suffice to say, our current AP is pretty strong in these two locations.

Find the average signal strength of all APs at a particular location:

select SSID, AVG(SIGNAL) AS AVGSIGNAL
FROM [dbo].[wifilog]
WHERE LOCATION='sysadmin'
GROUP BY SSID
ORDER BY AVGSIGNAL DESC

Similar to the first query, you can drill down by a particular location and see the top AP’s seen at that location. This is useful, but what we’re really looking for is the least used channels at a certain location.

Get a channel utilization chart

select LOCATION,CHANNEL,SSID,AVG(SIGNAL) AS AVGSIGNAL
FROM [dbo].[wifilog]
GROUP BY LOCATION,CHANNEL,SSID
ORDER BY AVGSIGNAL DESC

The above gives us some pretty useful information. We can see here that channels 8 and 11 have several entries, and only one device in range of our scans is on channel 6. Using some critical thinking, it’d indicate that channel 3 might be a good choice should we want to add a new AP to this environment. Lets massage the query a bit to see if that’s confirmed by our other data:

select LOCATION,CHANNEL,SSID,AVG(SIGNAL) AS AVGSIGNAL
FROM [dbo].[wifilog]
WHERE CHANNEL in (1,2,3,4,5,6)
GROUP BY LOCATION,CHANNEL,SSID
ORDER BY AVGSIGNAL DESC

At first blush, one might be enthusiastic about channel 3 given the fact it’s not used as the main carrier frequency in any of our entries. Be careful, though, for WiFi channels have some pretty strong overlap:

As this graphic shows, each wifi “Channel” is merely just a 5MHz swatch of the 2.4ghz ISM band. WiFi signals have a 22Mhz bandwidth, so realistically there’s only 3 channels one can use in an environment without any fear of interference or overlap. Because of this, one needs to take into account not only the channel but also the signal strength of potentially interfering access points.

In our case at Stack Exchange, there are just so many APs utilizing so many channels that we’ve ultimately decided to go with a Cisco controller-based access point layout, which will dynamically change channels based on signal conditions in realtime. For those of us at home, this is way too expensive of an option for most. Sadly, we’ll just have to take these data queries and give it our best shot.

I hope this blog series helps you a bit with your next WiFi installation. In summary:

• Antenna choice matters when you’re trying to cut through interference or travel long distances.
• Most residential building materials will not diffuse wireless signals to an appreciable amount unless you’re talking about very far distances, (i.e. trying to use your laptop on the third floor at the far side of your house when the AP is in the basement, for instance.)
• Be aware of electronics in your home that might share the 2.4GHz radio spectrum; they can seriously affect your wireless transfer rate and signal strength.
• Apps like Vistumbler can catalog used channels in your environment and you can then use this data to find a quiet spot in the spectrum.

As always, I welcome your comments and criticisms, below. Also, feel free to share any specific SQL queries you used that might help glean even more information from the datasets you’ve gathered!

It has been my experience that many people simply buy a wireless access point, plop it down squarely next to their home cable/dsl modem, and assume that’s all they have to do to maximize their WiFi experience. Oh, were it to be so simple! I’d like to take a few minutes of your time today to cover some of the basics of what WiFi is, what it is and is not capable of, and how you as a SysAdmin or a home user can do a bit of detective work to help ensure your WiFi experience is less prone to issue.

RF Basics

Let’s take a moment and talk about Radio-Frequency Radiation. RF is a form of non-ionizing radiation where waves of energy radiate from a source and follow a predictable pattern based on the transmitter power and antenna. Radio waves are measured based on the size of the wave, and how frequently the wave oscillates. The frequency is measured in Hertz (Hz), or cycles per second.

Wavelength is the distance the radiation travels before it completes a single cycle. As we are mentioning travelling, we need to know the speed, right? This, my friends, is the speed of light.

C = f * λ , which translates to: Speed of Light = Frequency * Wavelength OR, if you're lazy, 300/Frequency in megahertz. Light travels approximately 300 million meters per second, we can drop a whole bunch of zeros from the equation and still be reasonably accurate.

WiFi signals operate on 2.4 gHz (2.4 billion cycles per second), and that means that one full wave travels around 12-13 centimeters before the waveform returns to its starting position relative to the axis in the graph. 802.11a and 802.11n operate on the 5gHz range, which would put the signal wavelength at 6 centimeters.

OK, but, why should I care about this when all I want to do is surf porn and play online games? The answer lies in the fact that if your antenna is not properly suited for these measurements, it won’t work that well. The antennas you get from your access point vendor are “suitable” but far from ideal.

Not many people realize it, but there is an aftermarket for antennas for access points. When people/companies buy commercial grade access points, they usually don’t include any antennas, as it’s assumed you’re going to get the proper antenna for your application.

So, what types of antennas are there and what are the differences?

Omnidirectional – These are the antennas that people are most familiar with. They will usually be oriented vertically, and radiate their signal on the horizontal plane in all 360 degrees. See below radiation chart which does a good job of visualizing how the energy travels out of a veritcal omni antenna.

Directional/Yagi – Directional antennas are designed to send a signal straight to a specific spot with pinpoint accuracy. If you’re trying to setup a WiFi link between your house and a neighbor down the street, you’d need a directional antenna. The Pringles Cantenna is an example of a homemade directional antenna. Commercial antennas more closely resemble old TV antennas that everyone seemed to have on their house back in the 20th century. The below radiation pattern does look a bit weird, but understand that the directional beam is designed to be highly selective of signals based on its relative orientation versus the target signal. This allows a directional antenna to receive and send to stations much further away than an omnidirectional antenna, which sends RF energy in all directions.

patch – Patch antennas are normally flat antennas that are designed to radiate in a forward direction extremely well, with the signal attenuating sharply at the periphery. The radiation pattern below does have some similarities to the directional/yagi radiation pattern, but its lobe is more rounded in the forward direction. The patch antenna type is a good choice when you want to direct most of your energy in a particular direction but don’t necessarily want the pinpoint accuracy of a yagi.

What blocks WiFi?

WiFi, operating in the 2.4ghz and 5ghz ranges, propogates in “line of sight.” Due to the short wavelength, the energy dissipates quicker if it is not channeled into a high-gain directional antenna. Consumer grade access points come with omnidirectional rubber duck antennas, which people usually orient vertically. If you look at the above radiation pattern, you’ll see that there is a void of energy directly above omnidirectional antennas when they are oriented horizontally.

All matter will attenuate RF energy to some extent as it passes along. The question on many people’s minds is what are the worst places you can install a wireless router or access point? Believe it or not, most materials in the home are not capable of attenuating your WiFi signal to a noticeable degree. In order for WiFi signals to be blocked effectively, they need to move through several layers of dense material in order to shed the energy required to become unusable. Some antenna manufacturers will quote how well the radio waves will propogate from a given antenna, as shown here for one of Cisco’s branded antennas:

The density of the materials used in a building’s construction determines the number of walls the signal must pass through and still maintain adequate coverage. Consider the following before choosing the location to install your antenna:

• Paper and vinyl walls have very little affect on signal penetration.
• Solid and pre-cast concrete walls limit signal penetration to one or two walls without degrading coverage.
• Concrete and wood block walls limit signal penetration to three or four walls.
• A signal can penetrate five or six walls constructed of drywall or wood.
• A thick metal wall causes signals to reflect off, causing poor penetration.
• A wire mesh spaced between 1 and 1 1/2 in. (2.5 and 3.8 cm) acts as a harmonic reflector that blocks a 2.4-Ghz radio signal. (NOTE: as a commenter below further explains, this type of mesh is common in plaster walls from the 1940s as well as in stucco applications.)

I once ran into an issue with a friend who had his wireless router installed in the basement, next to his cablemodem. He was having sporadic connectivity issues in a second floor room and asked me to come help diagnose. Sure enough, his room was directly above the wireless router, two residential floors below, and given that traverse and the location of his room in relation to the radiation pattern, there wasn’t enough RF energy propogating up into that location. The short-term answer for the problem was to orient his access point antennas horizontally, so that the radiation pattern is then set on its side, covering a wider swath of his house.

In a blog post to come, I will show you some methods you can use to help properly locate your access point and also help you decide which frequency your access point should operate on. Stay tuned, and as always your comments,criticisms and suggestions are always welcome!

In order for system administrators to do their job well, particularly in a tech company, they need to know a lot of what is going on. This is because just about everything is done on the systems we control.

Lets look at some of examples of things system administrators probably need to know, and why they need to know them:

Example 1: Upcoming projects

In order to make sure we have enough capacity in servers, network, backups, etc we need to know what is incoming. If we don’t, it can be a lot more difficult to be prepared and that can slow things down.

Example 2: How a service or code works

System administrators are generally the first line of troubleshooting. In order to troubleshoot a problem, we need to know what is being done before we can trying to figure out why it isn’t working. We also need to monitor and backup the system, knowing how it works tells us what details to monitor and what data needs to be backed up.

Example 3: How important something is to the company

Resources are always limited. Although you want minimum standards of things like monitoring and backing up, time and money is limited — system administrators need some context for setting priorities. This can also help with figuring out an appropriate level of security.

Example 4: What people do

System administrators control access, so we need to have an idea of what sort of access people should have. We also need to know the best people to talk to when their is a problem or there is maintenance to do.

Knowing without Being Nosey

If we accept that system administrators really do need to know quite a bit of what is going on, then system administrators need to figure out how to do this without being nosey:

Definition of NOSY
: of prying or inquisitive disposition or quality : intrusive

The challenge is to have a good handle on what is going on, without prying or being intrusive. Part of the difficulty is that this is a two step process:

1. Find a way to sincerely not be nosey
2. Don’t come off as being nosey

These two steps are not easy, and require constant vigilance. If you have mastered them, then you probably don’t have to ask for information most of the time — information will be given to you and you will be invited to be part of the process.

Getting to that point is tricky, and I certainly don’t claim to have all the answers. In part it requires the cooperation of the other people in your company, but if we hold up our end of the bargain it goes a long way.

So what can system administrators do?

1. Don’t be nosey. Make it clear that knowing this information is not for your entertainment or to make you feel special, rather it is to enable you to better do your job.

2. Make things easier. Although sometimes doing your job requires you to get in the way, you should strive to add requirements because it makes things for everyone easier in the long run, not to exert power or justify your existence. If you don’t need to actually add a process or make things more difficult — then don’t. In many companies you want to be conservative with how much process you add.

3. Be consistent. Telling one person on the system administration team should be as good as telling everyone. Once you get involved, document, backup, and monitor everything. If your team is consistent it goes towards developing a reputation of making things easier for everyone.

4. Be respectful. If you work with great people, making sure things are good on the system side should be about being thorough. It is an SA’s job to think about that side of things full time, but it doesn’t mean the people you work with didn’t already think about it, or are being dumb if they didn’t.

5. Know your place. If you are invited into the process of a new project, keep in mind why you are there. If you have a really good idea out of your area of expertise try to share it tactfully. But if you are there mostly to listen, then try to mostly just listen.

In the end I think knowing everything that is going on, without being nosey, is pretty difficult. Most of us at some time or another have probably failed at some of the things I listed — it takes some honest self evaluation to find where you are falling short. Any readers have ideas for how to stay apprised of everything without being nosey?

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

• More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
• Licensing Costs
• Owned vs Rented Assets
• How many cloud “servers” or instances you would need vs real hardware
• Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

On more than one occasion I have been asked “how do I get a job in IT?” This question could easily be shrugged off and relegated to the canned answers pile; phrases I have personally uttered in the past include “go get some certifications then send out resumes” or “play with computers for a few years and then apply.” These answers really are a cop-out on my part so I’d like to take a few minutes to apply some serious thought to the query and offer my thoughts.

What People Will Expect of You

While most people would say that having a large breadth and depth of knowledge in computers is the primary requirement for being employed in IT, I would disagree with this. There are a handful of very critical attributes that you will find in any IT Rockstar:

• Good Problem Solving Skills
• Critical Thinking
• Strong work ethic
• Ability to handle “Burst Stress”

Problem Solving is possibly the most important attribute any good IT team member will have. Problem solving is the core of our vocation, when you think about it. IT is tasked with solving problems that the other employees aren’t able to handle. You’ll also find that in many workplaces, the IT guys are the ones helping to solve other complex problems in the enterprise because of the problem solving skillset they demonstrate. Critical Thinking is another important skill to have when talking about technical jobs. Oftentimes the problem I’m trying to solve does not have an easy-to-see solution (sometimes, the cause itself is not immediately apparent.)

Do you like working long hours for little praise or thanks? That’s pretty much working IT in a nutshell. Performing server maintenance usually needs to wait until after hours, which generally means you’ll be putting in a full 8 hours for that workday, then many more hours that night to do the after-hours downtimes. Couple this with “burst stress” and you’ve got a recipe for gray hair. Burst Stress is common in jobs like police work or firefighting, where you have long periods of low stress followed by short bursts of extremely high stress situations.

What do you want to do?

There are a handful of disciplines in IT, each with their separate purposes. What follows is a breakdown of the individual disciplines and what that type of work entails. Regardless of what path you choose, you’ll likely start in an IT Generalist role, and many people stay in that role their entire career. I myself genuinely enjoy being a generalist; it means that any given day I might be working on any number of problems that aren’t the same old issues over and over again.

IT Generalist — This is often the least respected but most useful role in IT. As a generalist, you are a Jack of All Trades. You’re expected to understand not only Desktop and Server Support, but also have a useable knowledge set in Telephony, Networking, Backup and Disaster Recovery, and Security. When you start your new IT career, you’ll usually start as a quasi-apprentice in a generalist role. Once you’ve “earned your stripes” in a few disciplines, you’ll be more educated and ready to move to a specialization, if that’s your pleasure.

Desktop Support — The Desktop Support team will generally be tasked with maintaining the software and equipment that other people in the company utilize to get their jobs done. This job is very demanding from a customer service perspective, as the person doing this job will oftentimes be assisting users who are already annoyed that their machines aren’t working the way they’d like! Skillset wise, Desktop engineers need to be highly fluent in whichever desktop operating system your company uses, as well as any applications the company makes use of. Generally speaking, you’re going to need to know Windows and Office like the back of your hand if you want to get a job like this in the majority of businesses.

Server Support/Systems — The Systems or “Server Support” team has similar requirements as Desktop Support when it comes to strong knowledge sets of operating systems and software. The jobs diverge when you look at the “back-of-the-house” operations that a Systems team is often assigned. Knowledge of datacenter operations (power distribution, hot row/cold row, rack positioning) is essential in this role if your organization has more than just a “closet in the back with a few servers.” Server Support is also usually in charge of all backend/utility systems in the organization, including directory authentication, mail and groupware, and administrating backups.

Networking — This group focuses mostly on interconnecting sites and equipment together. To get into this group, you better be prepared to prove you know the OSI model like the back of your hand. This is a tough group to coast in; TCP/IP can be fickle under wavering hands. Usually, a particular company will standardize on one vendor for their equipment, and you’ll be expected to know the operating system for that particular vendor. If you’re looking to get into the networking team at a new company, you should likely know Cisco and Juniper OS’s (Cisco IOS/ASA-OS, JunOS, etc.) Even if the company does not use either of these major brands, having knowledge of how other platforms operate can assist you in doing the job. Networking also is susceptible to the above mentioned “Burst Stress”, since if a site-to-site link is down, it’s possible that hundreds of people are sitting waiting for you to fix the problem.

Security — To be a member of the security team is to be alternately loved and hated by differing groups in your organization, often different groups at different times. Security generally finds themselves tasked with doing audits of internal data security and setting standards to help achieve compliance with data security standards. Computer forensic applications are the toolbox of the IT Security team; to do the job it will help to be seasoned at data analysis/data mining and event correlation. Security is usually the team tasked with defending audits as well, so it helps if you have good verbal communication skills. Finally, Security is sometimes tasked with coming up with the Disaster Recovery and Business Continuity Plans for a business. In some cases, other departments handle this, but in my experience this has been a Security-type role.

Database Administration — DBA is a tough specialization to be hired in. Brent Ozar has a great blog post covering this, so I won’t go in to too much detail but instead refer you to Brent Ozar’s excellent blog post series linked here. Suffice to say, you need to be able to speak SQL and make it sound like Shakespeare before you can roll with the titans like Mr. Ozar.

Telephony — I’m hesitant to group Telephony into IT, but many times it falls under the moniker of IT, so we’ll cover it here. Telephony is the specialization that handles telephones and telephony technologies. You’re going to want to know the details of how signaling works in PSTN networks, things like TDM, FXO,FXS, E&M, CAS, and a whole lot of other crazy acronyms. Depending on the size of the organization, you might also need to have some cross-pollination with the Networking skillset.

In a future blog post, I will cover what I consider the “elephant in the corner” when it comes to hiring in IT. Remember, it’s not always what you know, but who you know that gets you hired. As always, your comments and criticisms are welcome/encouraged; I’d like to hear what the community feels on this topic.

Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.

At Stack Exchange our use case for virtualization is growing. We are not going to run our core QA web servers and database servers using virtualization for performance reasons, but we do host things such as our monitoring system, blogs, domain controllers, and VPN servers.

Our collection of assorted services continues to grow, and with it so does our need to expand our virtualization setup. Currently in our main data center we have 3 VMWare ESX servers. But as we expand, how are we going to handle this growth?

Why Use Virtualization?

Virtualization at its heart is an abstraction layer between the hardware and the operating system. I have always had mixed feelings about this because operating systems, in theory, are supposed to provide all the hardware abstraction and inter service protection you need. However, system administrators have to live in the real world, and this just isn’t the case.

This layer of abstraction, as any abstraction, has performance implications. This in short is why we are not using it for our core QA service. The advantages of this abstraction layer however are tantalizing:

• Live migration (vMotion in VMWare terms)
• Running multiple operating systems (i.e. Windows and Linux) on the same hardware
• Easier to get full utilization of hardware resources by moving VMs around

These advantages and others exist because of this abstraction layer. From a pure systems perspective, the allure of virtualization is to deliver us from many of the hardware constraints when we design systems and go about our day to day tasks. Operating systems become modular to the hardware, and with modularity comes flexibility and agility. Flexibility and agility come from the lifting of constraints and are perhaps some of the most desirable qualities in a system. However, does virtualization deliver on this promise of flexibility?

The Joy of Commodity Hardware

As Wikipedia defines it:

“Commodity computing (or Commodity cluster computing) is to use large numbers of already available computing components for parallel computing … commodity computing done with commodity computers as opposed to high-cost supermicrocomputers or boutique computers.”

Today the commodity computer is your standard x64 computer with some varation of one or a couple cores, SAS or SATA spinning disks or SSDs, and some memory. You can debate where to draw the line in this, for instance some might call servers from Dell “specialized” servers where as boxes built from parts at Newegg are not. However, I consider all this commodity hardware because they are essentially variations on the same design — basically better versions of your home computer. The opposite of this is specialized hardware. With specialized hardware, there are major differences between vendors and they generally their own OS or a specialized variant of an operating system.

So what is the joy of commodity hardware? In my mind it is that it delivers on some of the same ideals that we want virtualization — modularity and flexibility. When you design for commodity hardware your servers are essentially interchangeable parts. They can be reused for other things and easily upgraded or replaced with newer versions as computing evolves. It also generally scales in a linear fashion, when you need more power, you just add more boxes.

Specialized hardware on the other hand has the advantage of being more well suited and optimized for its particular task. With this optimization though comes with the cost of lost modularity. Probably the most common example of specialized hardware in many data centers are SANs. They are the ultimate performers when it comes to storage, but you are likely not going to easily swap out your SAN and it can become a central constraint you design around.

Virtualization and Centralized Storage are Best Friends

With VMWare and many forms of virtualization, many of the features are designed to expect shared storage which generally comes in the form of a SAN. This relationship can be seen on the business side of things as well — EMC, one of the largest players in storage, is also the primary holder of VMWare.

Because the traditional virtualization infrastructure is designed around shared storage, the flexibility provided by virtualization comes in conflict with the flexibility of commodity hardware. That doesn’t mean shared storage can’t provide its own form of flexibility, but in my mind, these two are at odds with the traditional virtualization architecture. One of my main concerns is that over time the specialized hardware will weigh us down.

Virtualized Clusters to the Rescue?

If we can have the best of both worlds, it seems to me that it is going to come in the form of a virtual cluster. I first learned about these from a short presentation I saw by Tom Limoncelli about Ganeti. Ganeti is a console for managing virtual clusters built on top of Xen or KVM that is used at Google for some of their internal systems. The idea essentially is that you have a rack of commodity machines with many VMs per machine and still have the ability to do live migration. Using DRDB (think raid 1 across multiple machines) allows for features like live migration without shared storage.

VMWare also offers an appliance called the VMWare vSphere Storage Appliance (VSA) which seems like it might also deliver some of the features you normally only get with a SAN without the SAN — but this doesn’t seem to be the traditional VMWare design.

Virtualized clusters seem like they will give us a lot of the flexibility we want from virtualization while also allowing us to stick with commodity hardware. Writes across network RAID will be slower because they need to be commited to the mirror, but not all VMs would need to have this enabled, and I don’t think performance is our primary concern when it comes to our use of virtualization.

What Will We Go With?

Like when we tried to figure out what to do about storage, I don’t think this is a choice we can make over night. Virtual clusters are very appealing to me, but we will need to take them for a spin and learn what the limitations are. Centralized storage doesn’t sit well with the ideals and promises of commodity computing, but as I said before, system administrators need to operate in the real world with real constraints — so a SAN might be the best solution for us.